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Technology  Transition  Handbook 


1 .  Purpose.  This  Handbook  sets  forth  guidelines  and  procedures  to  facilitate 
the  smooth,  effective  and  timely  transition  of  technologies  and  technical 
information  developed  by  Science  and  Technology  (S&T)  projects  from  the 
Joint  Science  and  Technology  Office  (JSTO)  to  the  Joint  Program  Executive 
Office  for  Chemical  and  Biological  Defense  (JPEO-CBD)  Joint  Project 
Manager  (JPM). 

2.  Applicability.  This  handbook  applies  to  the  JPEO-CBD  and  the  JSTO.  It 
also  applies  to  other  agencies  preparing  and  submitting  science  and 
technology  efforts  for  transition  into  the  Joint  Chemical  and  Biological  Defense 
Program. 

3.  Releasability.  This  Handbook  is  approved  for  public  release;  distribution  is 
unlimited.  Department  of  Defense  (DoD)  components  (to  include  the 
combatant  commands),  other  federal  agencies,  and  the  public  may  obtain 
copies  of  this  manual  through  the  JPEO-CBD  or  the  JSTO. 

4.  Effective  Date.  This  Handbook  is  effective  upon  receipt. 
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Introduction 


The  purpose  of  this  handbook  is  to  provide  guidance  and  establish  procedures  to 
facilitate  the  smooth,  effective,  and  timely  transition  of  technologies  and  technical 
information  developed  by  Science  and  Technology  (S&T)  projects  from  the  Joint 
Science  and  Technology  Office  (JSTO)  to  the  Joint  Program  Executive  Office  for 
Chemical  and  Biological  Defense  (JPEO-CBD)  Joint  Project  Manager  (JPM). 
Transition  into  acquisition  programs /programs  of  record  will  be  accomplished  by 
"leveraging  the  best  technology  available  from  both  government  and  commercial 
sources"  to  "rapidly  transition  technology  into  new  material  systems"  to  reduce  or 
resolve  warfighter  capability  gaps.  A  key  enabler  for  evolutionary  acquisition  and 
reduced  cycle  time  is  to  have  technology  that  is  sufficiently  mature  to  be  fielded  in 
a  relatively  short  time. 

This  document  provides  guidance  for  the  transition  of  all  defense  science  and 
technology  base  systems  and  components,  Defense  Technology  Objective  (DTO) 
Programs,  and  Advanced  Concept  Technology  Demonstrations  (ACTD)  executed 
within  or  transitioned  into  the  Chemical  Biological  Defense  Program  (CBDP).  It 
also  encompasses  S&T  programs  and  unsolicited  proposals  from  other  DoD 
agencies  and  non-governmental  organizations  assimilated  into  the  JSTO 
developmental  process. 

Back^ound 

Prior  to  the  CBDP  Implementation  Plan  (Under  Secretary  of  Defense  for 
Acquisition  Technology  and  Logistics  (USD  (AT&L)),  April  2003),  advisors  to  the 
Office  of  the  Secretary  of  Defense  concluded  that  the  CBDP  S&T  base  was  not 
necessarily  aligned  with  the  warfighter’s  needs,  contributing  to  a  perception  of  too 
little  return  on  investment.  As  a  result,  the  use  of  management  incentive  tools  was 
encouraged  and  the  development  of  processes  was  initiated  that  would  facilitate 
the  transition  of  technologies  out  of  research  and  development  to  the  acquisition 
community.  The  CBDP  Implementation  Plan  has  established  the  framework  for 
the  JPEO-CBD  and  the  JSTO  to  develop  processes  to  positively  effect  the 
transition  of  technologies  from  S&T  to  acquisition  in  order  to  meet  the  needs  of 
the  warfighter.  This  handbook  documents  those  processes  and  tools  necessary  to 
accomplish  that  goal. 
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Scope 

This  handbook  addresses  the  processes  for  transition  of  technology  and  technical 
information  from  the  JSTO  to  the  JPEO-CBD,  from  DoD  and  other  government 
agencies  (i.e.  the  Defense  Advanced  Research  Projects  Agency  (DARPA), 
Department  of  Homeland  Security  (DHS),  etc.),  as  well  as  non-governmental 
agencies  (industry  and  academia)  through  the  JSTO  to  JPEO-CBD.  This 
handbook  will  address:  1)  the  technology  transition  process  supporting  the 
development  of  an  acquisition  strategy  by  the  JPEO-CBD  JPM,  2)  the 
development  of  S&T  program  transition  exit  criteria,  Technology  Readiness 
Levels  (TRL),  and  the  documentation  of  such  in  a  Technology  Transition 
Agreement  (TTA),  3)  the  process  for  Technology  Readiness  Evaluations  (TRE) 
and  Technology  Readiness  Assessments  (TRA),  and  4)  the  conduct  of  the 
Transition  Quarterly  Review  (TQR). 

This  handbook  is  intended  to  be  used  by  the  JPMs,  the  JSTO  Capability  Area 
Project  Officers  (CAPOs),  the  Joint  Test  and  Evaluation  (T&E)  Executive,  and 
the  Joint  Requirements  Office  for  Chemical,  Biological,  Radiological,  and  Nuclear 
Defense  (JRO-CBRND)  as  appropriate  for  the  documentation  of  T&E 
capabilities  in  the  transition  process  and  the  conduct  of  the  TQR. 

Responsibilities 

The  Implementation  Plan  for  the  Management  of  the  CBDP  assigns  two  agencies 
with  primary  responsibility  for  managing  the  transition  of  emerging  technologies 
in  the  CBDP:  the  JSTO  and  the  JPEO-CBD.  Figures  1  and  2  depict  the  JSTO  and 
JPEO-CBD  organizations. 
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JSTO 


Defense  Threat  Reduction  Agency  -  Chemical 
and  Biological  Defense  Directorate 


Figure  1.  JSTO  Organization  Chart. 

Projects  are  executed  under  the  CAPO  within  the  appropriate  S&T  Division  of 
the  JSTO.  Within  the  JSTO,  the  Defense  Threat  Reduction  Agency  Chemical 
Biological  Transition  Directorate  (DTRA-CBX)  is  tasked  with  managing  and 
facilitating  the  transition  process.  The  focus  of  DTRA-CBX  is  to  assess  and 
transition  mature  Chemical  and  Biological  (CB)  technologies  to  support  future 
acquisition  and  current  product  improvement.  The  JSTO  seeks  to  provide 
appropriately  mature  technologies,  which  can  be  inserted  into  JPEO-CBD 
acquisition  programs.  Within  the  JPEO-CBD,  the  Science,  Technology,  and 
Analysis  Directorate  is  responsible  for  coordinating  and  facilitating  technology 
transition.  For  purposes  of  this  document,  DTRA-CBX  will  be  referred  to  as 
JSTO  and  the  JPEO-CBD  Science,  Technology,  and  Analysis  Directorate  will  be 
referred  to  as  JPEO-CBD  hereafter. 
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Figure  2.  JPEO-CBD  Organization  Chart. 

The  following  paragraphs  outlining  technology  transition  responsibilities  are 
derived  from  the  CBDP  Implementation  Plan. 

JOINT  REQUIREMENTS  OFFICE  FOR  CBRN 
DEFENSE 


The  responsibilities  of  the  JRO-CBRND  pertaining  to  technology  transition  are  as 
follows: 

1)  Serve  as  the  principal  Joint  Staff  representative  for  CBDP  issues  and  focal  point 
for  coordination  with  the  Services. 

2)  Develop /maintain  the  Chemical,  Biological,  Radiological,  and  Nuclear  (CBRN) 
Defense  Joint  Overarching  Operational  Concept  and  Architecture.  Integrate 
relevant  portions  of  other  Joint  Operational  Architectures. 

3)  Represent  the  Services  and  Combatant  Commanders  in  the  DoD  Joint 
Capability  Integration  and  Development  System  (JCIDS)  and  act  as  the  Joint 
advocate  for  coordinating  and  integrating  Services  and  Combatant  Commander 
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approved  CBRN  defense  operational  capabilities,  to  include  Homeland  Defense 
and  Civil  Support  requirements.  Coordinate  and  manage  the  CBRN  defense 
requirements  document  approval  process  to  include  approving  Service  and 
Combatant  Command  validated  joint  requirements  documents  along  with 
Service/ Combatant  Command  specific  approved  annexes,  as  per  the  latest  version 
of  the  Chairman,  Joint  Chiefs  of  Staff  Instruction  (CJCSI)  3170.01  and  the  Joint 
Requirements  Oversight  Council  (JROC)  Memorandum  163-02  dated  9 
September  2002. 

4)  Coordinate  with  the  Services,  JPEO-CBD,  JSTO,  DARPA  and  the  Joint  T&E 
Executive  for  the  CBDP  and  the  Office  of  the  Director  for  Operational  Test  and 
Evaluation  (DOT&E)  to  ensure  joint  medical  and  non-medical  CBRN  defense 
materiel  requirements  are  effectively  evaluated  in  developmental  test  and 
evaluation  (DT&E)  and  operational  test  and  evaluation  (OT&E)  in  accordance 
with  applicable  directives,  including  Food  and  Drug  Administration  (FDA) 
directives  for  FDA-regulated  materiel. 

5)  Lead  the  development  of  the  DoD  CBDP  Program  Objectives  Memorandum 
(POM)  with  JPEO-CBD  and  JSTO  S&T  support 


JOINT  SCIENCE  AND  TECHNOLOGY 
OFFICE  FOR  CHEMICAL  AND  BIOLOGICAL 
DEFENSE  (JSTO) 


The  responsibilities  of  the  JSTO  pertaining  to  technology  transition  are  as  follows: 

1)  Manage  and  integrate  chemical  and  biological  science  and  technology  (CB  S&T) 
programs. 

2)  Generate  the  supporting  Receiver  Operating  Characteristic  (ROC)  curves  and 
equipment  attributes  for  the  spider  chart  used  in  the  TRA. 

3)  Develop  and  execute  CB  S&T  programs  approved  by  Assistant  to  the  Secretary 
of  Defense  for  Nuclear,  Chemical  and  Biological  Defense  (ATSD  (NBC))  in 
response  to  Joint  and  Service  needs  and  capabilities  requirements. 

4)  Coordinate  closely  with  JPMs  to  develop  the  6.2  and  6.3  S&T  programs  (see 
Appendix  I  for  definitions  of  the  Research,  Development,  Test  and  Evaluation 
(RDT&E)  budget  activities)  to  ensure  S&T  meets  acquisition  program  needs. 
Ensure  effective  transition  between  CB  S&T  programs  and  JPEO-CBD 
acquisition  programs  by  conducting  TRAs  and  TREs.  Jointly  develop  CB  S&T 
strategy  roadmaps  and  CB  S&T  Research,  Development  and  Acquisition  (RDA) 
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plans. 


5)  Integrate  with  Joint  Chiefs  of  Staff  (JCS)/JRO-CBRND  for  CB  S&T 
requirements. 

6)  Chair  the  TQR  in  cooperation  with  the  JPEO-CBD,  JRO-CBRND,  and  the 
Joint  T&E  Executive  to  effectively  manage  the  S&T  transition  process. 

JOINT  PROGRAM  EXECUTIVE  OFFICE  FOR 
CHEMICAL  AND  BIOLOGICAL  DEFENSE 
(JPEO-CBD) 


The  responsibilities  of  the  JPEO-CBD  pertaining  to  technology  transition  are  as 
follows: 


1)  Provide  centralized  program  management  and  Joint  Service  CBDP  acquisition 
program  integration  for  all  assigned  Joint  CBDP  non-medical  and  medical  efforts 
to  include  planning  guidance,  direction,  control,  and  support  necessary  to  ensure 
systems  are  developed  in  accordance  with  DoD  acquisition  guidance. 

2)  Establish  Technology  Readiness  Levels  (TRL)  in  conjunction  with  JSTO  and 
participate  in  the  JSTO  chaired  TQR  with  JRO-CBRND  and  the  Joint  T&E 
Executive  to  identify  opportunities  for  transition  of  CB  S&T  technologies  to 
acquisition. 

3)  Establish  exit  criteria  that  must  be  met  in  order  for  the  program  to  transition. 

4)  Determine  the  metrics  and  attributes  of  the  CB  equipment  to  be  used  in  the 
development  of  the  ROC  curve  and  spider  chart. 

5)  Ensure  interagency  cooperation  and  timely  transition  of  technologies  to  future 
development  programs  in  order  to  reduce  development  cycle  times. 

6)  Provide  technical  and  functional  integration  across  assigned  medical  and 
physical  science  (non-medical)  programs. 

7)  Ensure  integration  with  related  DoD  materiel  programs  required  for  force 
health  protection. 

8)  Coordinate  with  JRO-CBRND,  the  JSTO,  and  the  Joint  T&E  Executive 
regarding  conduct  of  a  semi-annual  JPEO-CBD  JPM/JSTO  CAPO  alignment 
meeting.  Assess  S&T  program  status  and  ensure  S&T  programs  are  synchronized 
and  funded  for  continued  development  and  transition. 
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9)  Coordinate  with  JSTO  to  jointly  develop  CB  S&T  Strategy  Roadmaps,  an  RDA 
Plan,  and  to  conduct  TRAs  and  TREs. 


JOINT  TEST  AND  EVALUATION  (T&E) 
EXECUTIVE 


The  responsibilities  of  the  Joint  T&E  Executive  pertaining  to  technology 
transition  are  as  follows: 

1)  Establish,  in  consultation  with  the  other  Services’  T&E  Executives,  a  common 
set  of  processes  and  standards  for  the  conduct  of  CBDP  T&E 

2)  Identify,  in  coordination  with  JPEO-CBD  and  JSTO,  CBDP  infrastructure 
capability  and  methodology  gaps,  requirements,  and  strategies  for  T&E  that  are 
associated  with  efforts  focused  on  transition  to  the  JPEO-CBD. 

3)  Establish,  review,  and  supervise  CBDP  T&E  procedures. 

4)  Oversee  CBDP  T&E  associated  with  RDA  in  addition  to  combat  and  training 
development  programs. 

5)  Coordinate  on  the  content  and  adequacy  of  the  Test  and  Evaluation  Strategy 
(TES)  supporting  the  TTAs. 
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Technology  Transition 
Process 


The  primary  intent  of  this  handbook  is  to  focus  on  transition  of  CBDP 
technologies  to  advanced  development  acquisition  activities  (Research, 
Development,  Test,  and  Evaluation  (RDT&E)  Activity  6.4).  This  handbook  does 
not  describe  the  entire  acquisition  life  cycle  process.  Elements  of  the  overall 
process  will  be  discussed  to  provide  perspective  and  points  of  reference.  This 
handbook  assumes  that,  as  a  minimum,  a  valid  Initial  Capabilities  Document 
(ICD)  exists  for  new  technologies  currently  in  development  or  that  a  Mission 
Needs  Statement  (MNS)  and  a  Joint  Operational  Requirements  Document 
(JORD)  exists  for  legacy  S&T  programs. 

The  DoD  preference  is  to  provide  capability  to  the  warfighter  through 
evolutionary  acquisition.  When  a  program  uses  an  evolutionary  acquisition 
strategy,  each  increment  of  capability  has  a  specific  set  of  parameters  with 
thresholds  and  objectives  appropriate  to  the  increment. 

In  an  evolutionary  approach,  the  CBDP  acquisition  strategy  describes  the  initial 
increment  of  capability  (i.e.,  the  initial  deployment  capability),  and  how  it  will  be 
funded,  developed,  tested,  produced,  and  supported.  The  CBDP  acquisition 
strategy  supported  by  this  transition  process  incorporates  similar  planning  for 
subsequent  increments  and  identifies  the  approach  to  integrate  and / or  retrofit 
earlier  increments  with  later  increments. 

If  the  capability  documents  do  not  allocate  increments  of  capability  (leading  to  full 
capability)  to  specific  program  increments  consistent  with  an  evolutionary 
approach,  the  JPM  and  CAPO  should  work  closely  with  the  JRO-CBRND  and  the 
user/ sponsor  (warfighter)  to  determine  whether  an  evolutionary  acquisition 
approach  will  serve  the  user/ sponsor  needs.  Where  necessary  and  acceptable  to 
the  user/ sponsor,  the  capability  documents  should  be  modified  by  the  JRO- 
CBRND  to  include  an  evolutionary  acquisition  approach.  The  technology 
transition  process  described  here  addresses  the  flexibility  of  the  CBDP  process  to 
meet  this  approach. 

The  ICD  and  the  Technology  Development  Strategy  (TDS)  guide  this  effort.  The 
TDS  will  form  the  basis  of  the  acquisition  strategy  prepared  for  the  Milestone 
(MS)  B/  technology  insertion  opportunity.  During  the  development  of  the  TDS, 
a  TES  is  developed  through  the  efforts  of  the  material  developer  (JPEO-CBD), 
the  requirements  office  (JRO-CBRND),  and  the  science  and  technology  provider 
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(JSTO)  and  coordinated  with  the  Joint  T&E  Executive.  These  processes  are 
described  later  in  the  Handbook. 

Multiple  TRAs  may  be  necessary,  but  are  not  required,  before  the  user  and 
developer  agree  that  a  proposed  technology  solution  is  affordable,  militarily  useful 
and  based  on  mature  technology.  The  TDS  shall  be  reviewed  and  updated  upon 
completion  of  each  technology  spiral  and  developmental  increment.  Updates  shall 
be  approved  by  the  JSTO  and  JPEO-CBD  to  support  follow-on  increments. 

The  initial  capability  in  an  evolutionary  strategy  represents  only  partial  fulfillment 
of  the  overall  capability  described  in  the  ICD;  successive  technology  transition 
efforts  continue  until  all  capabilities  have  been  satisfied  or  until  no  technology  can 
be  identified  to  meet  a  capability  need.  In  an  evolutionary  acquisition,  the 
identification  and  development  of  the  technologies  necessary  for  follow-on 
increments  continues  in  parallel  with  the  acquisition  of  preceding  increments, 
allowing  the  mature  technologies  to  more  rapidly  proceed  into  System 
Development  and  Demonstration  (SDD).  Each  increment  of  an  evolutionary 
acquisition  program  shall  have  an  associated  Milestone  Decision  Authority  (MDA) 
approved  TDS.  The  TDS  is  the  responsibility  of  the  JPM  and  is  required  prior  to 
MS  A. 

The  goal  of  the  DoD  S&T  community  is  to  develop  technology  as  quickly  as 
possible  and  then  successfully  transition  it  to  an  acquisition  program  of  record. 
This  S&T  structure  must  be  flexible  and  agile  in  order  to  respond  to  diverse  and 
complex  challenges.  S&T  should  provide  for  a  competitive  pipeline  that  forces 
competition  and  gets  technology  transitioned. 

During  Technology  Development,  the  JRO-CBRND  prepares  the  Capability 
Development  Document  (CDD)  to  support  program  initiation,  refines  the 
integrated  architecture  and  clarifies  how  the  program  will  lead  to  a  joint 
warfighting  capability.  The  CDD  builds  on  the  ICD  and  provides  the  detailed 
operational  performance  parameters  necessary  to  design  the  proposed  system.  A 
MS  B  decision  follows  the  completion  of  Technology  Development. 

CBDP  S&T  projects  intended  for  a  MS  B  program  initiation,  S&T  projects  funded 
in  RDT&E  Advanced  Technology  Development  (Budget  activity  6.3),  and  S&T 
projects  identified  for  programs  in  Applied  Research  (Budget  Activity  6.2)  require 
a  TDS  and  a  TES.  These  efforts  now  require  a  TTA.  Next,  a  TRE  Plan  is 
developed  in  order  to  ensure  that  the  technology  will  meet  JPM  requirements  and 
is  ready  to  transition  as  well  as  to  ensure  test  capabilities  and  methodologies  exist 
to  adequately  evaluate  the  technology.  TRLs,  ROC  curves,  and  metrics  are  then 
established.  If  necessary,  a  TRE  is  conducted  to  gather  data  to  support  the  next 
step  in  the  process,  the  TRA.  If  the  TRA  concludes  that  the  technology  has  met 
the  criteria  outlined  in  the  TTA,  the  technology  may  successfully  transition  to  the 
acquisition  activity.  Figure  3  outlines  how  the  technology  transition  process  and 
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documentation  timeline  coordinates  with  the  acquisition  and  capabilities 
development  processes. 


Technology  Transition  Documentation  Timeline 
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Figure  3.  CBDP  Technology  Transition  Process/Documentation  Timeline. 


The  CBDP  will  coordinate  with  other  Departments  and  agencies  along  with 
academia,  industry,  and  international  partners  (Figure  4)  to  identify  promising 
technologies  that  can  be  incorporated  into  existing  or  new  programs  to  fill  JRO- 
CBRND  identified  capability  gaps.  These  technologies  may  evolve  from  DARPA, 


Figure  4.  Overview  of  the  CBDP  Technology  Transition  Process. 

the  Technical  Support  Working  Group  (TSWG),  laboratories,  research  centers, 
academia,  or  foreign  and  domestic  commercial  sources.  The  CBDP  transition 
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process  will  reduce  the  risks  of  introducing  these  technologies  into  the  acquisition 
process;  and  promote  coordination,  cooperation,  and  mutual  understanding  of 
technology  issues.  In  some  cases,  an  increment  of  capability  may  be  demonstrated 
during  a  TRA  in  which  case  a  Limited  Utility  Assessment  (LUA)  may  be 
conducted  to  further  reduce  transition  time  to  the  program  of  record.  The 
conduct  of  S&T  activities  shall  not  preclude,  and  where  practicable,  shall  facilitate 
future  competition  to  provide  capabilities  to  the  CBDP  acquisition  process. 


14 


Management  Tools  for 
Technology  Transition 

In  the  past,  many  S&T  programs  concluded  but  technologies  did  not  transition 
because:  1)  the  acquisition  program  was  not  ready  to  receive  the  technology,  2) 
there  was  no  funding  to  support  the  transition,  or  3)  there  was  no  acquisition 
program  requirement  to  support  the  transition. 


The  management  tools  and  process  outlined  in  this  handbook  assures  mature 
technologies  transition  to  acquisition  programs,  close  coordination  between  JSTO, 
JPEO-CBD,  JRO-CBRND,  and  the  T&E  Executive  occurs  in  a  timely  manner. 
Additionally  adequate  funding  is  programmed  for  technology  transition,  T&E 
capability,  and  T&E  methodology  development  and  planning.  Capability 
documents,  beginning  with  the  Initial  Capabilities  Document  (ICD),  must  clearly 
articulate  the  capability  and  the  need  for  the  technology.  Coordination  is  essential 
to  ensure  that  the  JSTO  S&T  programs,  critical  to  the  execution  of  a  program  of 
record,  are  aligned.  Figure  5  depicts  the  CBDP  Alignment  Process. 
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Figure  5.  CBDP  Alignment  Process. 
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The  primary  tools  used  to  facilitate  the  CBDP  technology  transition  process  are: 

•  Technology  Development  Strategy  (TDS); 

•  Test  and  Evaluation  Strategy  (TES); 

•  Technology  Readiness  Level  (TRL); 

•  Manufacturing  Readiness  Level  (MRL); 

•  Technology  Transition  Agreement  (TTA); 

•  Technical  Readiness  Evaluation  (TRE); 

•  Technology  Readiness  Assessment  (TRA); 

•  Receiver  Operating  Characteristic  (ROC)  Curve;  and 

•  Transition  Quarterly  Review  (TQR). 

Figure  6  outlines  the  CBDP  technology  transition  products,  responsible 
organizations  and  required  timelines. 
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Figure  6.  CBDP  Technology  Transition  Responsibility  Matrix. 
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TECHNOLOGY  DEVELOPMENT  STRATEGY 


The  TDS  provides  rationale  for  adopting  either  an  evolutionary  acquisition 
strategy  or  a  single-step-to-full-capability  strategy.  For  an  evolutionary  acquisition, 
either  spiral  or  incremental,  the  TDS  will  include  a  description  of  the  capability/ 
system  in  which  the  technology  is  integrated  and  a  preliminary  description  of  how 
the  program  will  be  divided  into  technology  spirals  or  developmental  increments. 
The  TDS  outlines  a  program  strategy,  incorporating  overall  cost,  schedule,  and 
performance  goals  for  the  total  research  and  development  program.  Also  included 
are  exit  criteria  for  demonstration  in  a  relevant  environment,  for  the  first 
affordable  increment  of  technology.  Further,  the  TDS  contains  a  description  of 
tests  required  to  ensure  that  the  goals  and  exit  criteria  for  the  first  technology 
spiral  demonstration  are  met. 

The  acquisition  framework  incorporates  a  Technology  Development  Phase 
focused  on  the  development,  maturation,  and  evaluation  of  the  technologies 
needed  for  the  capability  under  consideration.  Phase  activities  concentrate  on 
maturing  those  technologies  (consistent  with  recommended  TRLs)  and 
demonstrating  readiness  to  proceed  with  program  initiation.  The  Technology 
Development  Phase  ends  when  the  MDA  determines  that  technologies  are 
sufficiently  mature.  This  determination,  along  with  the  satisfaction  of  other 
statutory  and  regulatory  requirements,  supports  program  initiation.  The  TDS  is  a 
statutory  requirement  (Sec  803,  Pub.L,  107-314)  documented  in  the  Defense 
Acquisition  Guidebook  (DAG)  which  focuses  specifically  on  the  activities  of  the 
Technology  Development  Phase.  Where  feasible,  the  TDS  should  also  discuss 
activities  associated  with  the  post-program-initiation  phases  of  the  planned 
acquisition. 

The  TDS  precedes  the  formal  Acquisition  Strategy  and  is  required  for  MS  A.  The 
TDS  is  updated  at  subsequent  milestones  and  subsumed  into  the  Acquisition 
Strategy.  If  the  Acquisition  Strategy  is  approved  at  MS  A,  the  TDS  may  be 
included  in  the  Acquisition  Strategy.  While  there  is  no  mandatory  format  for  the 
TDS,  Public  Law  107-314,  Section  803,  requires  the  following  minimum  content: 

•  A  discussion  of  the  planned  acquisition  approach,  including  a  summary  of 
the  considerations  and  rationale  supporting  the  chosen  approach.  For  the 
preferred,  evolutionary  acquisition  approach,  whether  spiral  or 
incremental,  DoD  Instruction  5000.2  requires  the  following  details: 

•  A  preliminary  description  of  how  the  program  will  be  divided  into 
technology  spirals  and  development  increments; 

•  The  limitation  on  the  number  of  prototype  units  that  may  be  produced 
and  deployed  during  technology  development; 
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•  How  prototype  units  will  be  supported;  and 

•  Specific  performance  goals  and  exit  criteria  that  must  be  met  before 
exceeding  the  number  of  prototypes  that  may  be  produced  under  the 
research  and  development  program. 

•  A  discussion  of  the  planned  strategy  to  manage  research  and  development. 
This  discussion  must  include  and  briefly  describe  the  overall  cost, 
schedule,  and  performance  goals  for  the  total  research  and  development 
program.  To  the  extent  practicable,  the  total  research  and  development 
program  should  include  all  planned  technology  spirals  or  increments. 

•  A  complete  description  of  the  first  technology  demonstration  or  TRE. 

The  description  must  contain  specific  cost,  schedule,  and  performance 
goals,  including  exit  criteria,  for  the  first  technology  spiral  demonstration. 

DoD  Instruction  5000.2  requires  that  each  increment  of  an  evolutionary 
acquisition  program  have  a  MDA-approved  TDS.  The  Instruction  also  requires 
that  the  TDS  be  reviewed  and  updated  upon  completion  of  each  technology  spiral 
and  development  increment  and  that  approved  updates  support  follow-on 
increments 

TEST  AND  EVALUATION  STRATEGY 


The  TES  is  an  early  T&E  planning  document  that  describes  the  T&E  activities 
starting  with  Technology  Development  and  continuing  through  SDD  into 
Production  and  Deployment.  Over  time,  the  scope  of  the  TES  will  expand  and 
evolve  into  the  Test  and  Evaluation  Master  Plan  (TEMP)  due  at  MS  B.  The 
development  of  the  TES  is  the  responsibility  of  the  projected  program  of  record 
material  developer  supported  by  the  JSTO  CAPO.  The  TES  is  coordinated  with 
the  Joint  T&E  Executive.  The  TES  is  part  of  the  TTA  and  is  reviewed  as  part  of 
the  TQR  process. 

The  TES  describes,  in  as  much  detail  as  possible,  the  risk  reduction  efforts  across 
the  range  of  activities  (e.g.,  Modeling  &  Simulation  (M&S),  DT&E  and  OT&E, 
etc.)  that  will  ultimately  produce  a  valid  evaluation  of  operational  effectiveness, 
suitability,  and  survivability  before  full-rate  production  and  deployment.  The  TES 
is  a  living  document  and  should  be  updated  as  determined  by  the  T&E  Working- 
group  Integrated  Process  Team  (WIPT)  during  the  Technology  Development 
Phase.  The  development  of  the  TES  will  require  early  involvement  of  testers, 
evaluators,  and  others  as  a  program  conducts  pre-system  acquisition  activities. 
These  personnel  will  provide  the  necessary  expertise  to  ensure  nothing  is 
overlooked  in  laying  out  a  complete  strategy.  The  TES  should  be  consistent  with 
and  complementary  to  the  Systems  Engineering  Plan  (SEP). 

The  TES  describes  the  system  in  which  the  technology  is  to  be  integrated,  the 
process  for  technology  integration,  and  how  the  component  technologies  being 
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developed  will  be  demonstrated  in  a  relevant  environment  to  support  the 
program’s  transition  into  the  SDD  Phase.  The  TES  addresses  the  concept  of 
employment  and  operational  strategy  supporting  a  possible  early  operational 
assessment  of  the  technology.  The  TES  contains  hardware  and  software  maturity 
success  criteria  used  to  assess  key  technology  maturity  for  entry  into  SDD.  The 
TES  is  the  planning  document  used  to  begin  developing  the  entire  program’s 
T&E  Strategy,  and  includes  the  initial  T&E  concepts  for  Technology 
Development,  SDD,  and  beyond. 

For  programs  following  an  evolutionary  acquisition  strategy  with  more  than  one 
developmental  increment,  the  TES  should  describe  how  T&E  and  M&S  would  be 
applied  to  confirm  that  each  increment  provides  its  required  operational 
effectiveness,  suitability,  and  survivability  as  would  be  required  of  a  program 
containing  only  one  increment.  The  development  of  the  TES  establishes  an  early 
consensus  among  T&E  WIPT  member  organizations  on  the  scope  of  how  the 
technology  or  system  will  be  tested  and  evaluated,  with  particular  consideration 
given  to  needed  resources,  in  order  to  support  Planning,  Programming,  Budgeting, 
and  Execution  (PPBE)  process  activities.  The  TES  applies  to  S&T  efforts 
associated  with  development  of  subsystems,  components,  and  technologies 
supporting  program  acquisition  strategies. 

There  is  no  prescribed  format  for  the  TES,  but  it  should  include  the  following 
items,  to  the  extent  they  are  known: 

•  Introduction  and  objectives  of  the  system-specific  technical  and  operational 
evaluations  that  will  support  future  decision  events; 

•  System  description,  mission,  concept  of  operations,  and  major  performance 
capabilities  from  the  ICD.  Identify  new  technology  and  the  plan  to  identify 
associated  risk; 

•  Acquisition  strategy  concept  -  For  programs  following  the  preferred  evolutionary 
acquisition  strategy,  the  TES  should  describe  how  T&E  and  M&S  would  be 
applied  to  each  increment.  It  should  show  how  each  increment  would  ultimately 
provide  a  demonstrated  level  of  operational  effectiveness,  suitability,  and 
survivability,  and  meet  user  needs  with  a  measurable  increase  in  mission 
capability; 

•  Time-phased  threats  to  mission  accomplishment; 

•  Anticipated  concept  of  operations,  including  supportability  concept; 

•  Technical  risk  reduction  testing,  including  any  new  or  critical  technologies 
identified  in  the  TDS; 

•  Anticipated  component  and  sub-system  developmental  testing  that  begins  after 
MS  A; 

•  T&E  strategy  for  SDD; 

•  Critical  operational  and  live  fire  (if  appropriate)  issues; 

•  Scope  and  structure  of  the  operational  and  live  fire  evaluations; 

•  Likely  sources  of  required  data; 
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•  Major  T&E  design  considerations; 

•  Hardware  and  software  maturity  success  criteria; 

•  T&E  schedule; 

•  Anticipated  M&S  used  for  future  system  evaluations;  and 

•  T&E  funding  estimates  in  enough  detail  to  permit  programming  and  budgeting. 

For  all  programs  which  the  Office  of  the  Secretary  of  Defense  (OSD)  T&E 
maintains  oversight,  the  program  manager  or  leader  of  the  concept  development 
team,  with  the  T&E  WIPT  providing  support,  must  submit  the  DoD  Component- 
approved  TES  to  OSD  for  staffing  and  approval  before  MS  A.  Early  involvement 
of  testers  and  evaluators  will  ensure  a  better  product  and  will  expedite  the 
approval  process,  as  issues  will  be  addressed  and  resolved  early  through  the 
Integrated  Product  and  Process  Development  (IPPD)  process. 

The  TES  should  be  submitted  45  days  prior  to  MS  A  so  that  an  OSD-approved 
document  is  available  to  support  the  decision.  The  TES  portion  of  the  TTA  for 
S&T  projects  supporting  programs  on  the  OSD  T&E  Oversight  List  is  approved 
through  the  Joint  T&E  Executive.  For  programs  not  on  the  OSD  T&E  Oversight 
List,  the  JPEO-CBD,  or  designated  representative,  approves  the  TES. 


TECHNOLOGY  TRANSITION  AGREEMENT 


The  revision  of  the  Federal  Acquisition  Regulation  (FAR)  DoD  5000  brought 
about  significant  changes  in  the  way  technology  transitions  from  the  technical 
developer  to  the  advanced  developer  (i.e.  the  JPM).  An  integral  part  of  this 
process  is  the  use  of  the  TTA  as  taught  by  the  Defense  Acquisition  University 
(DAU).  The  DAU  S&T  Manager’s  course  (STM  301  and  302)  advocates  the  use 
of  TTAs  to  document  the  commitment  of  the  requirements/resource  sponsor  (in 
the  case  of  the  CBDP,  this  is  the  JRO-CBRND),  S&T  activity  (JSTO),  and 
acquisition  program  sponsor  (JPM)  to  develop,  deliver,  and  integrate  a 
technology/ product  into  an  acquisition  program. 

The  TTA  process  can  be  articulated  from  the  “Technology  Pull”  or  “Technology 
Push  scenario”  (see  figure  7).  In  the  case  of  “Technology  Pull”,  the  JPM  conveys 
the  technology  need  to  the  JSTO  and  then  closely  coordinates  with  the  JSTO  to 
“build”  an  S&T  program  to  meet  that  capability  and  develop  the  TTA  to  support 
the  technology  development,  TES,  and  transition.  In  the  “Technology  Push” 
scenario,  the  JSTO  may  have  a  proposal  for  a  technology  that  may  not  satisfy  a 
current  requirement  but  may  provide  substantial  benefit  to  the  program  justifying 
an  unplanned  product  improvement  or  horizontal  technology  insertion  into  an 
existing  program  of  record.  The  JSTO  would  then  coordinate  with  the  JPM  to 
jointly  develop  and  finalize  the  proposal  and  subsequent  TTA.  Close  coordination 
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between  the  CAPO  and  JPM  ensure  that  the  TTA  is  developed  for  the  technology 
that  will  most  likely  meet  the  needs  of  the  warfighter. 


Technology  Pull 


CAPO/JPM 

LJ 

Final 

Agreement 

1  1 

Proposal  1 

Figure  7.  CBDP  Technology  Pull  and  Technology  Push. 


The  TTA  is  a  Memorandum  of  Agreement  (MO A)  between  the  JSTO  (technology 
developer)  and  the  JPM  (intended  receiver  of  a  technology  or  capability 
development).  The  TTA  is  the  primary  vehicle  for  the  transition  process  of 
transitioning  6.3  technologies,  and  in  special  cases  6.2  technologies  or  technical 
information,  from  the  JSTO  scientific  developer  to  the  appropriate  JPM.  A  TTA 
documents  the  program  exit  criteria  of  the  JSTO  to  develop,  deliver,  test,  and 
evaluate  a  technology/ product  for  an  acquisition  program.  The  TTA  provides  a 
clear  assignment  of  responsibilities  of  what  each  party  provides  in  order  for 
transition  to  occur.  A  TTA  is  developed  for  each  JSTO  S&T  project  in  which  the 
next  phase  is  to  be  executed  is  by  the  acquisition  activity  (JPM).  For  the  CBDP, 
TTAs  are  applicable  to  all  JSTO  Physical  Science  and  Medical  6.3  programs  and 
some  6.2  programs  with  identified  technologies  or  technical  information  products 
but  do  not  apply  at  the  project  level.  To  clarify,  several  6.2  projects  may  be 
conducted  concurrently  leading  to  a  single  technology  that  will  transition  through 
a  6.3  effort.  For  example,  a  project  for  a  collector,  a  sensor,  and  a  method  of  data 
processing  would  culminate  in  a  6.3  prototype  for  a  biological  agent  detection 
system.  The  TTA  identifies  the  “target”  6.4  program  of  record  to  which  the  6.2  or 
6.3  technologies  are  intended  to  transition.  TTAs  are  required  for  all  6.3  CBDP 
programs,  with  a  few  exceptions.  One  possible  exception  is  a  6.3  program 
established  for  tests  that  complement  the  development  of  a  particular  technology 
but  is  not  by  itself  solely  for  that  technology  (i.e.  a  6.3  project  to  validate 
environmental  test  methodologies  that  are  applicable  to  various  detection  systems 
under  development).  Funding  for  6.3  programs  with  potential  technologies 
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lacking  TTAs  may  be  withheld,  by  the  Director,  JSTO,  until  a  valid  TTA  is 
provided.  For  programs  initiated  post  MS  A,  the  TTA  performs  the  function  of 
the  TDS  and  TES. 


There  are  two  key  factors  in  the  development  of  the  TTA:  1)  exit  criteria  that  must 
be  met  for  the  program  to  transition  and  2)  a  determination  of  the  metrics  and 
attributes  of  the  CB  equipment  to  be  used  in  the  development  of  the  ROC  curve 
and  spider  chart  within  development  processes.  The  JPM  establishes  the 
technology  exit  criteria  so  that  advanced  planning  for  test  methodology 
development,  testing,  evaluation,  and  facilities  construction  can  be  programmed. 
The  JPM  establishes  the  metrics  and  attributes  necessary  for  the  generation  of 
ROC  curves  and  spider  charts.  The  JSTO  is  responsible  for  the  generation  of  the 
data  supporting  ROC  curves  and  sensor  attributes  for  the  spider  chart  used  in  the 
technology  assessment.  A  template  for  a  TTA  can  be  found  in  Appendix  H.  The 
TTA  development  process  is  depicted  in  figure  8. 


JRO 
CBRND 

ICD  CDD 
CPD 


Figure  8.  CBDP  Technology  Transition  Agreement  Process. 


TTA  Termination:  Once  signed,  a  TTA  will  be  terminated  only  if:  1)  the 
requirement  in  a  JRO-CBRND  source  document  is  superseded  or  deleted,  2)  the 
requirement  for  that  technology/ capability  no  longer  exists,  3)  the  requirement  for 
the  technology  has  already  been  met,  or  4)  no  technology  can  be  identified  within 
the  necessary  timeline  for  incremental  acquisition.  Funding  constraints, 
developmental  delays,  immaturity  of  technology  as  demonstrated  through  TRA,  or 
initial  failure  to  meet  exit  criteria  under  T&E  will  not  be  used  as  rationale  to 
terminate  a  TTA.  However,  the  intention  is  not  to  “perpetuate”  R&D. 

Transition  Exit  Criteria:  All  technology  to  be  transitioned  from  the  JSTO  to  the 
JPM  will  have  exit  criteria  established  to  which  the  technology  is  tested  and 
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evaluated  and  is  documented  in  a  TTA.  For  effective  and  timely  transition,  exit 
criteria  will  be  determined  upon  entry  to  MS  A.  Exit  criteria  will  be  obtained  and 
developed  from  paragraph  two  (2),  Required  Capability  and  paragraph  three  (3), 
Concept  of  Operations  Summary,  of  the  ICD.  Exit  criteria  attributes  should  be 
measurable  and  quantifiable  to  ascertain  if  the  technology  meets  program 
requirements.  In  the  event  that  exit  criteria  is  not  specified  or  well  defined  in  the 
ICD,  the  JSTO  and  the  JPM  will  develop  exit  criteria  to  evaluate  that  the 
technology  meets  program  requirements  prior  to  transition.  With  very  few 
exceptions  (i.e.  to  satisfy  an  Urgent  Needs  Statement  (UNS)  or  Operational  Need 
Statement  (ONS)),  all  technology  to  be  transitioned  to  the  JPM  will  be  evaluated 
to  ensure  it  meets  the  program  criteria. 

The  process  involved  in  the  identification,  development,  and  intra/interagency 
staffing  coordination  of  TTAs  is  as  follows: 

TTA  Coordination:  The  TTA  is  developed  and  coordinated  for  approval  by  an 
assistance  team  from  DTRA-CBX  and  the  JPEO-CBD  Science,  Technology,  and 
Analysis  Directorate.  Both  organizations  are  responsible  for  transition  within  the 
CBDP.  The  TTA  assistance  team  supports  both  the  CAPOs  and  JPMs  in  the 
construct  of  each  TTA  as  an  adjunct  element  of  the  respective  acquisition  activity 
and  S&T  program.  The  TTA  assistance  team  must  be  familiar  and  experienced  in 
the  respective  capability  areas  they  support. 

The  JSTO  (via  DTRA-CBX)  coordinates,  as  part  of  a  transition  team  supporting 
DTRA-CBT  (Physical  S&T  Division  Chief)  and  DTRA-CBM  (Medical  S&T 
Division  Chief),  a  list  of  S&T  projects  that  have  been  identified  as  potentially 
having  a  “product,  software,  or  information”  for  transition  that  will  require  a 
TTA.  DTRA-CBT  and  DTRA-CBM  Division  Chiefs  and  the  supported  JPM 
review  and  validate  the  lists  of  S&T  projects  that  require  TTAs.  The  TTA 
Assistance  Team  coordinates  between  the  CAPO  and  JPM  to  develop  a  TTA  in 
accordance  with  this  handbook  and  serve  as  subject  matter  experts  for  TTA 
development  supporting  the  CAPO/  JPM. 

Although  transition  of  6.2  projects  to  a  6.4-level  of  advanced  development  is 
uncommon,  it  is  possible;  all  6.2  projects  will  be  reviewed  to  determine  if 
transition  potential  exists.  TTAs  will  be  developed  for  6.2  projects,  applied 
research,  in  all  cases  where  the  focus  of  the  applied  research  effort  is  to  resolve  a 
capability  gap. 

Prior  to  signature,  the  draft  TTA  is  provided  for  coordination  and  concurrence  to 
the  following:  the  JPEO-CBD  (JPEO-CBD  Director  of  Technology  and  Product 
Director  for  T&E  Systems  Support  (PD-TESS)),  JPM,  DTRA-CBX,  DTRA-CBT 
and  DTRA-CBM  Division  Chiefs,  JRO-CBRND  and  the  Joint  T&E  Executive. 
This  coordination  process  serves  to  review,  edit  and  finalize  the  TTA  to  the  point 
where  it  is  ready  for  signature. 
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The  TTA  is  signed  by  the  performing  CAPO  and  the  JPM  of  the  Program  of 
Record  (POR)  targeted  for  transition. 

After  signature,  the  TTA  is  periodically  reviewed  for  currency  as  a  part  of  the 
TQR  process. 


TECHNOLOGY  READINESS  EVALUATION 
PLAN 


The  TRE  Plan  is  developed  by  JSTO  in  close  coordination  with  JPM  and  the  Joint 
T&E  Executive.  The  TRE  Plan  documents  the  strategy  for  evaluating  the  T&E 
capability  development  necessary  to  test  and  evaluate  both  S&T  and  commercial 
products  that  may  be  considered  for  transition  to  a  joint  program.  The  TRE  Plan 
should  be  prepared  in  draft  form  no  later  than  one  year  prior  to  the  scheduled 
TRE  and  should  be  coordinated  among  all  concerned  parties.  The  final  draft 
should  be  prepared  and  coordinated  no  later  than  one  month  prior  to  completion 
of  the  development  effort.  The  TRE  Plan  will  be  part  of  the  MS  B  (or  MS  A 
process  in  the  case  of  medical  products)  approval  process  and  will  be  reviewed  for 
sufficiency,  quality,  and  adequacy.  The  TRE  Plan  will,  at  a  minimum,  include  and 
describe  the  following  areas  in  detail: 

•  Program  description 

•  Projected  transition  date 

•  Key  Performance  Parameters,  thresholds  and  objectives 

•  Projected  specific  capability  dates 

•  Current  status  of  the  program  to  include  fiscal  year  funding  levels 

•  Risk  analysis  and  mitigation  plan 

•  Timeline  with  milestones  and  key  events 

•  TRLs  and  MRLs 

•  Integration  strategy  of  product  into  the  JPM  Acquisition  program 
(alignment) 
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Technology  Transition 
Assessment  Tools  and 
Processes 

TECHNOLOGY  READINESS  LEVELS 


TRLs  are  measures  of  technical  maturity  and  form  the  basis  of  the  technology 
assessment.  The  MDA  will  consider  the  recommended  TRL  when  assessing 
program  risk.  The  JPEO-CBD,  executed  through  the  appropriate  JPM,  has  the 
lead  responsibility  to  establish  TRLs  for  technologies,  components  and  systems 
focused  on  transition  to  existing  programs  of  record.  The  JSTO  is  responsible  for 
assessing  the  maturity  of  those  technologies,  systems,  and  components  against  the 
TRLs  and  within  the  scope  of  the  supporting  TTA  through  the  TRA  process.  In 
other  words,  the  JSTO  recommends  the  TRL  and  the  JPM  assigns  the  TRL. 

TRLs  will  be  assigned  to  all  components  and  subcomponents  of  the  technology 
being  evaluated.  The  overall  TRL  for  a  system  is  determined  by  the  lowest 
assigned  TRL  for  the  components  and  subcomponents  within  the  system.  A 
guide  to  assigning  TRLs  can  be  found  in  Appendix  A.  TRL  definitions  are  located 
in  Appendix  B. 

The  use  of  TRLs  to  manage  the  integration  of  emerging  technologies  into  product 
development  efforts  has  been  found  to  dramatically  improve  program  success 
rates  (General  Accounting  Office  (GAO)  report  “Best  Practices:  Better 
Management  of  Technology  Development  Can  Improve  Weapon  System 
Outcomes”  (GAO/NSIAD-99-162).  The  report  identifies  three  key  knowledge 
points  that  directly  effect  product  development  risks,  cycle  times,  and  costs:  1) 
when  a  match  is  made  between  a  customer’s  requirements  and  the  available 
technology;  2)  when  the  product’s  design  is  determined  to  be  capable  of  meeting 
performance  requirements  and  3)  when  the  product  is  determined  to  be 
producible  within  cost,  schedule,  and  quality  targets  (Figure  9).  To  realize  their 
full  benefit,  TRL  assessments  are  best  performed  at  the  first  knowledge  point, 
prior  to  transition  into  product  development. 

Figure  9  can  be  compared  to  the  defense  acquisition  framework  found  in  DoD 
Instruction  5000.2  (Figure  10).  Knowledge  point  1  is  roughly  equivalent  to  MS  B 
in  this  framework. 
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Figure  9.  Key  Knowledge  Points  in  the  Acquisition  Process. 
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Figure  10.  Defense  Acquisition  Framework. 


TRLs  for  both  components  and  entire  systems  are  validated  during  TRAs.  DoD 
Instruction  5000.2  establishes  a  requirement  for  TRAs  for  Acquisition  Category 
(ACAT)  ID  and  ACAT  1AM  programs  prior  to  MS  B  and  C. 


MANUFACTURING  READINESS  LEVELS 


MRLs  are  metrics  used  to  assess  the  ability  of  the  industrial  base  to  mass  produce 
the  technology  that  is  to  be  transitioned  to  advanced  development  based  on 
current  industrial  manufacturing  processes  and  capabilities.  As  defined  by  the 
DoD  TRA  Deskbook,  “MRLs  are  measures  used  to  assess  system 
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engineering/ design  process  and  maturity  of  a  technology’s  associated 
manufacturing  processes.”  MRL  definitions  can  be  found  in  Appendix  G. 

The  JPM  has  the  lead  responsibility  of  assigning  MRLs.  If  requested  by  the  JPM, 
the  JSTO  CAPO  will  conduct  an  MRL  assessment.  When  the  JPM  elects  to 
conduct  an  MRL,  sufficient  time  will  allotted  so  as  to  provide  it  at  least  90  days 
prior  to  the  transition/MS  B  review.  The  TRA  panel  will  use  the  MRL  evaluation 
criteria  to  evaluate  the  maturity  of  the  technology  to  be  transitioned  to  ensure  that 
it  is  mature  enough  to  meet  the  JPM’s  needs  and  is  manufacturable  and  affordable 
in  the  quantities  required  to  meet  fielding  goals  and  timelines. 


EQUIPMENT  METRICS  AND  ATTRIBUTES 


The  performance  of  CB  agent  equipment  will  be  characterized  by  a  number  of 
interrelated  parameters  (e.g.,  metrics  and  attributes).  These  metrics  and  attributes 
will  be  developed  by  the  responsible  JPM  and  reflect  the  performance 
characteristics  of  sensors,  protective  equipment,  decontamination  solutions,  and 
models  used  in  the  Joint  CB  Defense  Program.  For  purpose  of  illustration,  sensor 
equipment  will  be  discussed  in  this  handbook,  but  the  methodology  is  applicable 
in  all  CB  defense  commodity  areas.  Metrics  such  as  sensitivity,  probability  of 
correct  detection,  false  positive  rate,  and  response  time  which  will  be  developed  in 
a  sensor  ROC  curve  (See  Figure  11.).  Attributes  include  factors  such  as  weight, 
cost,  reliability,  maintenance,  and  logistic  factors  to  name  only  a  few.  The  Spider 
chart,  Figure  12,  will  be  generated  to  capture  the  metrics  and  attributes  of  a  desires 
CB  defense  technology.  The  TTA  will  specify  the  metrics  and  attributes  to  be 
used  in  the  development  of  the  technology. 


Key  Metrics 

*  Sensitivity 

*  Probability  of  detection 

*  False  positive  rate 
■  Response  time 


Other  Attributes 

*  Cost 

*  Power  consumption 

*  Reliability 

*  Ma  in  tena  nee;  Logistics 

*  Size  and  weight 
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Figure  A.  The  key  sensor  rn  etrics  andth  eir  relation  to  the  ROC  curve . 
Other  attributes  also  strongly  affect  the  utility  of  specific  sensors. 


low 


->  high 


False  Positive  Rate 


Figure  11.  Example  of  Receiver  Operating  Characteristic  (ROC)  Curve  for  a  CB  sensor 
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For  example  the  JPM,  and  the  JSTO  CAPO  will  define  the  ROC  curves  for  a 
sensor  to  include  time  as  the  independent  variable  and  have  integrated  levels  of 
detection  overlaid  on  them  (Figure  11). 

Sensor  metrics  will  relate  sensor  sensitivity  rate  to  false  positive  at  a  given 
detection  confidence  for  a  determined  response  time.  Therefore,  sensor  metrics 
to  be  developed  under  each  sensor  TTA  are; 

•  Sensitivity 

•  Probability  of  Detection 

•  False  Positive  Rate 

•  Response  Time 

In  addition  to  the  metrics,  sensor  attributes  will  be  developed  and  included  in  the 
sensor  TTA  for  development  in  the  6.3  program.  These  attributes  will  be  agreed 
to  in  the  TTA  and  may  include  some  or  all  of  the  following. 

Initial  cost  affects  how  equipment  is  employed  and  the  numbers  of 
sensors/equipment  employed.  Disposable  sensors /equipment  should  be  very 
inexpensive,  while  non-disposable  sensors/ equipment  deployed  with  units  on  the 
battlefield  could  cost  significantly  more.  In  contrast,  a  single  sensor  unit  for 
protecting  a  facility  from  external  attack  may  be  quite  expensive,  whereas  multiple 
sensors/ equipment  employed  for  internal  attacks  might  cost  less.  Depending  on 
performance  and  mission  requirements,  equipment  costs  could  change 
dramatically. 

Operating  cost  is  comprised  of  any  cost  incurred  after  the  initial  acquisition 
expenditure.  This  includes  both  logistic  and  maintenance  costs,  consumable 
supplies,  repair  parts,  and  operator  training.  Operating  cost  can  range  from  very 
low  for  disposable  sensors/equipment,  to  lifetime  costs  that  greatly  exceed  the 
initial  cost  of  the  equipment  for  more  paramount  sensors/equipment.  If  only  one 
sensor/piece  of  equipment  is  to  be  maintained,  a  higher  operating  cost  may  be 
more  tolerable  than  in  a  situation  where  large  numbers  of  sensors/equipment  are 
deployed. 

Power  consumption  is  critical  to  the  mission  as  it  effects  the  mission  profile  of 
the  equipment  and  the  concepts  of  employments.  For  force  protection  roles,  the 
equipment  should  typically  be  battery  powered.  A  disposable  sensor/piece  of 
equipment  should  have  a  very  low  power  requirement.  For  building  equipment 
systems,  an  AC  line  would  be  available  to  provide  power.  Power  consumption 
must  also  be  considered  in  light  of  mission  duration. 

Maintenance  consists  of  the  actions  taken  to  keep  materiel  in  a  serviceable 
condition  or  restore  it  to  serviceability. 
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Reliability  is  the  probability  that  an  item  will  perform  to  its  intended  function  for 
a  specified  interval  under  stated  conditions.  The  longer  the  sensor  performs 
without  experiencing  an  unexpected  failure  (i.e.  the  mean  time  between  failure 
(MTBF)),  the  better  the  reliability.  It  is  assumed  that  stated  routine  maintenance 
requirements  are  met. 

Ruggedness  is  the  ability  to  withstand  shock,  vibration,  and  exposure  to  harsh 
weather  conditions  and  even  some  effects  of  enemy  nuclear  weapons  (e.g. 
electromagnetic  pulse  (EMP)). 

Form  factor,  i.e.,  the  size,  weight,  and  shape  of  the  equipment,  is  of  particular 
concern  in  the  battlefield  role  where  sensors/equipment  are  frequently  moved. 
Man  portable,  small  sensors/ equipment  are  highly  desirable  in  this  role.  Small 
form  factor  is  normally  less  critical  from  the  facility  standpoint  because 
sensors/equipment  will  usually  remain  in  place. 

Environmental  considerations  are  the  set  of  guidelines  meant  to  protect  the 
environment,  the  military,  and  noncombatant  civilian  populations.  These  include 
issues  such  as  safe  disposal  of  reagents  and  used  consumables  to  excessive  noise 
and  laser  eye-safety.  These  can  have  a  serious  impact  on  equipment  acceptance. 

The  examples  here  are  focused  on  sensor  metrics  and  attributes;  however  this 
approach  is  also  intended  to  apply  across  the  spectrum  of  CBDP  capability  needs. 
The  JPM  is  responsible  for  the  definition  of  metrics  and  attributes  of  technology 
to  support  an  acquisition  strategy. 

These  metrics  and  attributes  will  be  developed  for  the  TTA  in  order  to  assist  the 
JPM  in  transitioning  technologies  appropriately.  Some  important  facts  to 
remember  when  developing  equipment  metrics  are  listed  below: 

•  Sensitivity  will  always  be  stated  with  the  probability  of  detection,  the  false 
positive  rate,  and  the  response  time. 

•  Equipment  testing  used  to  evaluate  equipment  performance  in  the  6.3 
process  must  occur  in  environments  in  which  the  sensor/ equipment  will 
operate  and  must  be  generated  with  different  levels  of  detection 
confidence  in  a  single  environment. 

•  ROC  curves  must  be  developed  for  sensors/ equipment  at  each  stage  of 
the  development  to  determine  readiness  for  the  next  developmental  stage. 

Equipment  metrics  and  attributes  derived  from  the  ROC  curves  and  testing  are 
then  graphically  represented  in  spider  charts  to  compare  sensors/ equipment  or  to 
judge  the  overall  equipment  performance  (See  Figure  12).  In  a  spider  chart,  each 
of  the  sensor  metrics  is  assigned  a  “leg”  on  the  chart,  with  better  performance 
moving  out  from  the  center.  The  performance  of  the  equipment  can  then  be 
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plotted.  The  spider  chart  is  then  used  as  a  means  of  comparison  between 
sensors/ equipment,  or  simply  as  a  means  to  judge  overall  efficacy  of  a  given 
sensor/ piece  of  equipment. 
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Figure  12.  Spider  Chart  Example. 


ROC  curves  (as  applicable)  with  spider  charts  will  be  included  in  the  TTA  and 
used  in  the  TRA  process  as  a  tool  to  determine  the  TRL  of  the  technology.  A 
copy  of  the  CB  Sensor  Standards  Study  can  be  found  on  the  JPEO-CBD 
Integrated  Digital  Environment  (IDE)  under  the  Technology  Directorate  section. 
The  study’s  findings  have  been  incorporated  in  this  document. 


TECHNOLOGY  READINESS  ASSESSMENT 


A  TRA  is  the  review,  conducted  by  an  independent  assessment  panel;  for  a 
specific  component  or  system  that  has  been  determined  to  have  met  the  criteria  in 
the  TTA  (see  Appendix  A  for  methodology  to  develop  criteria).  The  TRA  Panel 
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is  chaired  by  the  JSTO.  Membership  on  the  panel  shall  include  representation  by 
the  JPM  for  whom  a  technology  is  being  developed  in  the  TTA. 


A  TRA  is  conducted  before  each  MS  B  and  MS  C  event.  In  the  case  of  medical 
programs,  a  TRA  may  be  required  prior  to  MS  A,  as  the  majority  of  medical 
products  transition  at  MS  A.  Before  a  technology  attains  MS  B  status  and 
transitions  to  the  SDD  stage,  a  TRA  must  have  been  conducted  and  the 
technology  must  have  been  demonstrated  in  a  relevant  environment  (or  an 
operational  environment,  if  possible).  This  will  occur  approximately  sixteen  (16) 
weeks  before  the  designated  MS  review  date.  The  assessment  will  be  performed  at 
the  direction  of  JSTO  and  must  include  all  critical  technologies  identified  by  the 
JPM  and  can  include  additional  technologies  that  the  JSTO  CAPO  deems  critical. 
Critical  technologies  are  defined  as  those  technologies  the  program/ system 
depends  on  to  meet  capability  thresholds.  While  much  of  the  information  comes 
from  the  JPM,  the  actual  assessment  is  made  independent  of  the  JPM.  Figure  13 
describes  the  basic  TRA  process. 


Bringing  in  New  Technologies: 
Technology  Readiness  Assessments  (TRA) 
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TRAs:  Determine  State  of  Technology  Against  Warfighter  Requirements 
OUTCOMES:  Technology  Transition/Insertion  and/or  Procurement 


Figure  13.  Technology  Readiness  Assessment  Process. 


For  Hardware  systems  with  incremental  development  strategies,  each  successive 
incremental  design  improvement  will  require  a  TRA  to  be  conducted  for  that 
increment  before  the  program  receives  a  MS  B  or  MS  C  review.  The  spiral 
development  process  is  normally  used  in  software  development.  In  the  TRA 
process,  software  is  considered  an  integral  part  of  the  system  or  subsystem  in 


31 


which  it  operates.  As  such,  demonstration  of  technical  maturity  at  the  subsystem 
or  system  level  must  include  a  demonstration  of  the  associated  software. 

The  TRA  Deskbook,  published  by  the  Deputy  Under  Secretary  of  Defense  for 
Science  and  Technology  (DUSD  (S&T))  provides  an  overview  of  the  TRA  process 
and  its  relationship  to  the  acquisition  process. 


TECHNOLOGY  READINESS  EVALUATION 


TREs  are  paper  studies  and / or  field  and  laboratory  tests  used  to  gather  data  in 
support  of  a  TRA  on  specific  technologies  to  meet  JPM  program  requirements. 
TREs  are  managed  by  the  JSTO  DTRA-CBX  division.  DTRA-CBX  works 
closely  with  both  the  JPM  and  the  JSTO  CAPO  to  ensure  that  the  TRE  will  meet 
the  JPM’s  needs.  TREs  are  conducted  on  S&T  technologies  prior  to  transition  to 
an  acquisition  program  to  support  a  MS  decision  or  pre-planned  product 
improvement  (P3I).  Ideally,  the  TRE  should  be  accomplished  at  the  beginning  of 
the  Concept  Refinement  phase  to  determine  what  technology  already  exists  that 
may  possibly  satisfy  the  acquisition  program  requirements.  Data  collected  during 
a  TRE  is  used  to  determine  the  effectiveness  and  suitability  of  a  technology  to 
meet  program  criteria  set  forth  by  the  JPM.  JPMs  are  therefore  prepared  to 
transition  a  technology,  component  or  system  which  is  supported  by  a  TRA/TRE 
and  with  a  good  understanding  of  the  technical  risks  derived  from  the  maturity 
level  resolved  through  the  TRL. 


TRANSITION  QUARTERLY  REVIEW  (TQR) 


The  TQR  is  a  high  level  execution  review  of  the  efforts  necessary  to  transition  a 
technology  to  the  appropriate  JPM.  The  TQR  is  conducted  with  membership  of 
the  JPEO-CBD,  JSTO,  JRO-CBRND  and  the  Joint  T&E  Executive. 

The  purpose  of  the  TQR  is  to  make  recommendations  to  the  appropriate 
management/ execution  entity  to  insure  confidence  in  the  transition  of  acceptable, 
mature  capability  on  time  to  the  material  developer. 

The  authority  for  conducting  the  review  is  derived  from  the  USD  Memorandum: 
Implementation  Plan  for  the  Management  of  Chemical  Biological  Defense 
Program,  dated  22  Apr  2003.  The  memorandum  directs  the  TQR  team  to: 

•  Identify  candidate  S&T  technology  areas/programs  for  future  transition 
and  plan  for  current  transition. 
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•  Review  transition-testing  programs  and  plans  for  tests  and  test 
methodology  development. 

•  Report  on  transition  tests  conducted  and  the  results. 

•  Develop  future  year  program  transition  requirements. 

•  Review  status  and  currency  of  TTAs. 

•  Review  and  update  CBDP  alignment  charts. 
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Medical  Transition  Process 


Medical  S&T  falls  into  two  categories:  1)  drugs  (encompassing,  pre-treatments, 
vaccines  and  therapeutics)  and  2)  diagnostics  (which  include  detection  and 
identification  from  clinical  material).  A  fundamental  difference  between  medical 
acquisition  programs  and  other  DoD  acquisition  programs  is  the  decision  to 
pursue  the  development  program  while  the  effort  is  still  in  the  technology  base, 
well  in  advance  of  the  traditional  MS  B  transition  point  for  DoD  programs.  The 
Food  and  Drug  Administration  (FDA)  regulatory  process  requires  product  and 
manufacturing  process  definition  to  be  well  defined  prior  to  human  clinical  trials. 
Changes  in  a  product,  once  clinical  trials  have  begun,  may  negate  previously 
accomplished  trials  resulting  in  increased  cost  and  schedule  slip  due  to  having  to 
repeat  clinical  trials.  Medical  product  development  and  production  programs 
must  therefore  be  defined  early.  The  medical  program  utilizes  the  DoD  system 
acquisition  process  defined  in  the  DoD  5000  series  documents.  This  process 
provides  the  discipline  necessary  to  ensure  a  successful  development  program 
while  providing  the  flexibility  necessary  to  allow  integration  of  FDA  regulatory 
processes. 

Medical  S&T  drug  development  begins  with  a  requirement  to  counter  an  existing 
threat  to  the  warfighter.  An  ICD,  developed  by  JRO-CBRND,  should  have  been 
generated  outlining,  in  broad  terms,  the  capability  required  for  the  drug 
development.  Drug  research  and  development  begins  with  pre-clinical  (animal) 
testing.  Pre-clinical  testing  involves  a  rigorous  evaluation  process  of  the  drug 
characteristics,  its  stability,  efficacy  and  manufacturing  process.  During  this 
period,  the  JSTO  is  down  selecting  candidate  drugs  to  arrive  at  one  (1)  or  two  (2) 
candidates  that  have  demonstrated  promising  results.  At  this  stage  in  the  process, 
TRLs  for  the  associated  drug  will  have  been  established  by  the  JSTO,  with  JPM 
concurrence.  When  the  JSTO  determines  that  the  drug  is  ready  for  transition  to 
the  JPM  for  advanced  development,  the  JSTO,  in  conjunction  with  the  JPM,  will 
request  a  MS  A  decision  from  the  MDA.  If  milestone  decision  is  approved,  a 
complete  technical  package  on  the  drug  is  compiled  and  submitted  to  the  JPM  at 
the  time  of  transition.  This  package  may  become  part  of  the  JPM  investigational 
new  drug  (IND)  application  packet  that  is  submitted  to  the  FDA. 

The  JPEO-CBD  Chemical  and  Biological  Medical  Systems  (CBMS)  JPM,  in 
conjunction  with  a  Prime  Systems  Contractor  (PSC)  (if  applicable),  begins 
coordinating  with  the  JSTO  when  candidates  are  in  the  DTO  stage  to  gain 
technical  familiarity  with  the  program  and  to  ensure  that  advanced  development 
funding  is  aligned  appropriately  to  support  a  candidate  at  MS  B.  This 
coordination  also  allows  CBMS  manufacturers  to  gain  early  visibility  of  the 
product  candidate.  The  management  lead  for  the  program  shifts  to  CBMS  at  MS 
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A,  although  both  S&T  and  advanced  development  funds  may  be  used  during  the 
Technology  Development  stage.  This  allows  the  manufacturer  to  engage  with  the 
JSTO  early  in  the  process.  If  multiple  candidates  are  pursued,  down  selection 
occurs  no  earlier  than  the  end  of  Phase  1  clinical  trials  that  are  conducted  prior  to 
MS  B  and  program  initiation.  Once  the  program  has  transitioned,  CBMS,  in 
concert  with  a  PSC  (if  applicable)  will: 

•  Conduct  Phase  I,  II,  and  III  clinical  trials  as  required 

•  Produce  pilot  and  consistency  lots 

•  Conduct  definitive  animal  efficacy  studies 

•  Submit  the  necessary  regulatory  documentation  to  obtain  licensure  to 
include  the  Biologies  License  Application  (BLA)  or  New  Drug  Application 
(NDA) 

MS  B  will  be  conducted  once  safety  and  efficacy  data  are  available  from  the  Phase 
I  clinical  trial  and/ or  animal  studies.  MS  C  will  be  conducted  once  consistency 
lots  have  proven  manufacturability  in  the  case  of  a  Low  Rate  Initial  Production 
(LRIP)  decision  (for  some  programs)  and  licensure  for  a  MS  C  Full  Rate 
Production  decision. 

Medical  Diagnostics  S&T  development  follows  along  the  same  pathway  as  that  of 
drug  development  with  some  differences.  In  diagnostic  S&T  development,  pre- 
clinical  and  clinical  trials  are  conducted  without  the  need  of  an  IND  application. 
The  clinical  trial  phase  involves  analysis  of  sensitivity  and  specificity  of  the 
diagnostics  kits.  This  is  accomplished  by  the  JSTO  via  non-invasive  human 
clinical  testing.  Transition  to  the  JPM  may  occur  during  this  period  or  the  clinical 
testing  phase  may  be  jointly  managed  by  the  JSTO  and  the  JPM.  If  clinical  testing 
validates  the  diagnostics  kits  performance,  a  technical  package  will  be  compiled 
and  submitted  to  the  FDA  for  approval  and  fielding. 

In  general,  medical  technology  transition  begins  with  initial  DoD/FDA 
discussions  followed  by  S&T  development  and  then  a  MS  A  decision.  Subsequent 
work  revolves  around  process  development/manufacturing  request  for  proposal 
(RFP):  award  process  and  manufacturing  contract  development;  initiation  of 
product  development  and  manufacturing;  animal  safety  studies;  IND  submission; 
phasel-III  clinical  trials,  submission  of  the  BLA  or  NDA,  FDA  licensure,  and 
final  transition  to  the  warfighter.  TREs  to  support  JPM  CBMS  will  be  conducted 
on  an  as  needed  basis.  The  requirements  for  technology  transition  documentation 
(TDS,  TES  and  TTAs)  are  applicable  to  JPM  CBMS  programs.  Medical  TRL 
definitions  can  be  found  in  Appendix  E. 
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Role  of  T&E  in  Technology 
Transition 


Planning  for  CBDP  T&E  will  begin  at  the  earliest  stages  of  the  definition  of  user 
needs,  science  and  technology,  system  requirements,  development,  and  acquisition 
processes.  System  evaluators  participate  in  the  integrated  concept  team  (ICT) 
review  of  the  initial  requirements  documents  when  a  new  CBDP  system  or  new 
technology  is  being  considered  for  development. 

The  early  involvement  of  the  T&E  community  has  become  increasingly  critical  to 
ensure  adequate  data  to  support  milestone  decisions.  Involvement  of  the  T&E 
community  in  the  test  and  evaluation  strategy  and  coordination  of  the  TTA  is 
critical  to  overall  success.  In  order  to  establish  this  early  T&E  involvement,  the 
CBDP  T&E  Executive  management  funds  are  used  to  support  early  involvement 
of  T&E  in  the  technology  development  process. 

The  Joint  T&E  Executive  supports  and  assists  the  JSTO  and  JPEO-CBD  in  the 
same  manner  as  any  joint  program.  Responsibilities  include  CBDP  T&E  policy, 
oversight  and  T&E  issues  resolution  procedures.  The  Joint  T&E  Executive  will 
also  establish  and  review  CBDP  T&E  procedures  for  transition  efforts. 

The  T&E  Executive  must  ensure  that  the  T&E  methodologies  and  capabilities  are 
adequately  identified  in  time  to  support  TREs  of  transitioning  technologies.  In 
order  to  do  this,  the  T&E  Executive  provides  a  T&E  investment  strategy  and 
supports  JPEO-CBD  programs  in  the  POM  process  to  identify  and  fill  T&E 
capability  gaps  for  programs. 

The  T&E  community  independently  assesses  how  well  systems  perform 
technically;  how  well  the  system  fulfills  documented  requirements  and  whether 
systems  are  safe,  operationally  effective,  suitable,  and  survivable  for  their  intended 
use  in  military  operations. 

The  T&E  community  does  not  establish  system  test  criteria;  these  criteria  are 
obtained  from  requirements  documents  and  other  sources  reflecting  system  user 
needs,  priorities,  and  operational  concepts.  The  T&E  community  does  define 
adequacy  of  test  and  thus  the  T&E  capabilities  required  to  perform  testing.  The 
input  of  the  T&E  community  to  developing  test  technologies  along  with  system 
technologies  is  critical.  The  T&E  community  must  have  input  into  the  process  as 
well  as  clear  and  well  defined  guidance  about  how  the  system  is  expected  to 
perform.  The  evolutionary  acquisition  concept  challenges  the  requirements, 
acquisition,  sustainment,  and  T&E  communities  to  coordinate  closely  and 
continually  when  developing  and  testing  phased  or  blocked  programs  to  ensure 
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that  the  T&E  community  is  aware  of  what  will  constitute  a  useful  increment  of 
capability.  Only  with  this  knowledge  can  the  T&E  community  design  appropriate 
tests. 

The  T&E  community  supports  evolutionary  acquisition  by  being  continuously 
involved  in  the  acquisition  process,  beginning  with  integrating  T&E  issues  in  the 
concept  and  technology  development  phase.  JPMs  can  form  a  WIPT  to  assist 
with  T&E  issues.  A  WIPT  can  assist  a  pre-systems  acquisition  activity  (e.g. 
ACTD,  Advanced  Technology  Demonstration  (ATD),  or  Joint  Warfighter 
Experiment  (JWE)  that  is  likely  to  develop  into  an  acquisition  program. 
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Appendix  A  -  Assigning  TRLs 
within  the  CBDP 


The  methodology  presented  here  represents  a  framework  from  which  TRL 
assessments  can  be  developed  which  is  consistent  across  the  CBDP  community. 

A  disciplined  application  of  this  methodology  will  result  in  TRL  assessments  that 
are  credible  and  understandable  regardless  of  which  agency  or  group  conducts  the 
evaluation. 

The  assessment  of  TRLs  is  anything  but  a  trivial  process.  The  success  and  validity 
of  the  assessment  depends  on  extensive  knowledge  concerning  the  development 
of  the  system,  a  thorough  understanding  of  the  technologies  involved,  a  clear 
definition  and  understanding  of  the  assessment  purpose,  the  data  available,  and  a 
good  grasp  of  the  TRL  definitions.  Absent  any  of  these  elements,  the  assessment 
results  are  suspect.  The  methodology  presented  here  addresses  the  last  element, 
dealing  with  the  TRL  definitions,  by  proposing  a  set  of  readiness  variable 
descriptions,  and  a  process  for  applying  them,  that  can  be  used  to  guide  the 
assessment  to  a  defensible  and  repeatable  conclusion.  Continuing  to  develop  and 
tailor  those  definitions  to  specific  technology  areas  within  the  CBDP  arena  will 
only  increase  their  utility  and  improve  the  validity  of  future  TRL  assessments. 

By  their  nature,  TRL  assessments  are  somewhat  subjective  and  vulnerable  to 
differences  of  opinion  concerning  the  status  of  a  technology  with  respect  to  the 
TRL  definitions.  The  readiness  variables  mitigate  a  portion  of  this  subjectivity  by 
offering  some  objective  mileposts  that  further  refine  the  basic  definitions  and  can 
be  tailored  to  the  technology  area  of  interest.  Even  so,  the  process  will  continue 
to  involve  individual  judgments  that  will  always  be  subject  to  argument  or 
disagreement.  This  is  one  reason  why  TRL  assessments  should  be  conducted  by 
more  than  one  person.  A  group  of  stakeholders  will  bring  different  viewpoints 
and  opinions  to  the  process,  making  it  stronger  in  the  long  run.  The  process 
outlined  here  provides  a  way  to  structure  those  judgments  so  that  they  can  not 
only  be  defended  in  the  face  of  criticism,  but  also  ensures  they  address  the  needs 
of  the  decision  makers  involved. 

TRLs  are  primarily  a  risk  management  tool.  The  JPM’s  decision  to  use  a  new 
technology  in  system  development  carries  with  it  considerable  cost  and  schedule 
risk,  a  defensible  TRL  process  will  clarify  the  JPM’s  risk  mitigation  strategy.  If  the 
technology  is  too  immature,  program  costs  can  sky  rocket  and  schedules  can  be 
delayed.  GAO  found  in  their  review  of  23  defense  programs  that  where  new 
technologies  had  matured  to  at  least  TRL  6  or  higher,  cost  and  schedule 
performance  for  the  program  were  much  improved  over  cases  where  more 
immature  technologies  were  adopted.  TRLs  provide  a  structured,  disciplined 
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approach  to  assessing  maturity  and  a  common  framework  with  which  to  discuss 
maturity  with  program  managers  and  decision  makers. 

While  general  statements  concerning  acceptable  risk  versus  TRL  are  useful  at  a 
macro  level,  specific  decisions  on  what  constitutes  acceptable  risk  for  transition 
depend  on  the  program,  the  technology  involved,  and  the  decision  maker’s  risk 
profile.  The  middle  ground  of  TRLs  4—6  constitute  a  gray  area  where  risks  may 
be  acceptable  or  unacceptable  depending  on  the  specific  situation.  Managers  have 
to  weigh  all  the  decision  criteria,  including  the  TRL  of  the  technology  in  question 
to  arrive  at  a  decision. 

In  a  July  2001  memorandum,  the  DUSD  (S&T)  officially  endorsed  the  use  of 
TRLs  in  new  major  acquisition  programs,  calling  for  TRL  assessments  for  “critical 
technologies”  identified  prior  to  the  start  of  engineering  and  manufacturing 
development  and  production.  The  Defense  Acquisition  Guidebook  (DAG,  17 
October  2004)  discusses  this  requirement  in  general  and  provides  definitions  for 
TRLs  at  a  system  level. 

TRLs  1  to  3  generally  apply  to  technology  development,  and  levels  above  this  to 
the  maturation  of  design  application.  In  the  case  of  technology  development, 

TRL  1  represents  basic  science  research  and  TRL  3  is  the  point  where  the 
performance  attributes  critical  to  use  in  practical  applications  are  demonstrated. 

By  definition,  application  concepts  have  not  been  explored  in  any  detail  at  this 
stage. 

Differentiation  between  TRLs  4  and  5  represents  the  transition  from  laboratory  to 
’real  world’  demonstration.  In  the  case  of  a  control  system  component,  TRL  4 
might  be  exemplified  by  artificial  stimulation  of  response  from  the  component  (i.e. 
the  representation  of  the  system  of  which  the  component  is  part  remains  virtual). 
This  can  be  compared  with  TRL  5  where  the  test  component  is  demonstrated  to 
work  within  a  physical  realization  of  the  overall  system  (i.e.  any  stimulation  is  to 
the  external  system).  The  test  component  at  TRL  5  might  be  representative  of  the 
technology  or  design  proposed  for  the  intended  system  application,  however  the 
overall  demonstration  system  would  not  be  representative  (i.e.  other  physical 
elements  within  the  demonstration  would  not  replicate  the  fit  or  form  of  the 
intended  application). 

Above  TRL  5,  demonstration  is  of  system  prototypes  or  models  (representative  of 
form  and  function)  with  increasing  similarity  to  the  production  system  (TRL  8), 
culminating  in  completion  of  minor  fixes  on  the  final  article  at  TRL  9,  which  will 
typically  be  cleared  for  operational  use.  While  it  is  not  always  appropriate  to 
develop  a  technology  through  every  level,  the  risk  associated  with  'skipping’  levels 
must  always  be  balanced  with  the  cost  of  taking  a  more  controlled  'step  by  step’ 
approach. 
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The  TRL  definitions  contained  in  the  DAG  are  constructed  at  the  system  level 
and  are  intended  to  apply  to  both  hardware  and  software.  Unfortunately,  the 
application  of  these  definitions  to  other  than  hardware  can  prove  difficult. 
Recognizing  this  limitation,  and  taking  advantage  of  the  flexibility  contained  in  the 
DAG  language  allowing  supplementation  of  the  basic  definitions,  the  US  Army 
Communications  Electronics  Command  (CECOM)  developed  a  set  of  alternative 
software  definitions  (Appendix  C).  Although  there  are  some  in  the  software 
development  community  who  think  the  Army  definitions  may  be  too  restrictive  in 
places,  they  remain  the  only  published  attempt  to  apply  the  basic  definitions  to  the 
area  of  software  technology. 

Missing  in  the  standard  definitions  are  references  to  non-system  technologies, 
such  as  processes,  methods,  algorithms,  or  architectures.  Non-system 
technologies  can  be  of  even  more  interest  than  hardware  in  the  CB  defense  arena, 
where  algorithms  can  play  an  especially  important  role.  TRAs  of  these 
technologies  require  an  additional  set  of  definitions  that  address  how  their 
development  and  testing  proceed  (Appendix  D). 

The  DAG  leaves  open  the  option  to  tailor  the  standard  definitions  to  specific 
technology  areas.  This  flexibility  allows  organizations  to  develop  TRL  definitions 
that  reflect  the  unique  characteristics  and  requirements  of  specific  types  of 
technologies.  As  an  example,  Appendix  E  includes  TRL  definitions  that  relate  to 
drug,  vaccines  and  medical  equipment. 

A  major  difficulty  facing  any  organization  tasked  with  assessing  TRLs  is  the  fact 
that  the  definitions  blend  together  several  aspects  of  readiness  into  each  definition. 
Each  step  of  the  TRL  scale  is  defined  by  a  combination  of  the  level  of  knowledge 
about  the  technology,  the  degree  of  integration  achieved,  the  development 
environment,  and  the  level  of  testing.  This  can  lead  to  a  dilemma  of  trying  to 
decide  which  aspect  takes  precedence  when  assessing  the  maturity  of  a  candidate 
technology.  Take  for  example,  the  definition  of  TRL  8  found  in  the  DAG: 


TRL  8 

Technology  has  been  proven  to  work  in  its  final  form  and  under 
expected  conditions.  In  almost  all  cases,  this  TRL  represents  the  end  of 
tme  system  development.  Examples  include  developmental  test  and 
evaluation  of  the  system  in  its  intended  weapon  system  to  determine  if  it 
meets  design  specifications. 


The  definition  requires  the  user  to  make  judgments  concerning  the  physical 
maturity  of  the  system  (“final  form”),  the  development  environment  (“expected 
conditions”),  and  the  type  of  testing  conducted  (“developmental  testing”). 
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Lacking  specific  guidance  on  how  to  weigh  each  of  these  aspects  of  readiness, 
users  will  typically  default  to  basing  the  assessment  primarily  on  the  level  of  testing 
achieved  or  adopt  an  “all  or  nothing”  approach  that  requires  each  aspect  to  be 
achieved  in  order  to  assign  a  specific  TRL.  At  the  higher  TRLs  (7,  8,  and  9)  this 
approach  may  be  acceptable  since  the  criteria  are  mutually  supporting  (i.e., 
satisfying  one  criteria  usually  means  the  others  have  been  met  as  well)  and  easily 
recognized  milestones  such  as  developmental  or  operational  tests  characterize 
each  level.  Unfortunately,  our  focus  is  not  usually  on  technologies  this  mature. 
Where  managers  need  the  most  fidelity  in  applying  the  TRL  definitions  is  exactly 
where  the  most  ambiguity  exists  in  the  definitions  (TRLs  1  to  6).  Technologies  at 
these  levels  of  maturity  are  usually  of  most  interest  for  transition  or  investment, 
but  can  often  present  conflicting  pictures  of  maturity  (see  example  below). 

Ambiguous  definitions: 

A  technology  for  point  biological  agent  detection  has  been  tested  in  a 
laboratory  environment  using  what  are  still  considered  low  fidelity 
components  (not  necessarily  representative  of  final  form,  fit,  or  function) 
and  agent  simulants.  As  part  of  an  examination  of  possible  emerging 
technologies  it  is  necessary  to  determine  the  system’s  TRL.  As  the 
process  of  assessing  the  technological  readiness  of  the  system  proceeds 
there  are  many  possible  sources  of  confusion.  First,  by  itself,  the  term 
“laboratory  environment”  could  be  consistent  with  TRLs  3,  4,  5,  or  6. 
Similarly,  “low  fidelity  components”  could  indicate  TRLs  3,  4,  or  even  5 
in  some  cases.  The  impact  of  the  fact  that  the  testing  has  been 
conducted  with  only  simulants  is  hard  to  discern  since  the  definitions  do 
not  refer  to  technology  specific  aspects  of  maturity. 


As  the  example  illustrates,  one  would  be  hard  pressed  to  determine  the  TRU  for 
this  system  using  only  the  standard  definitions  without  substantial  interpretation 
and  inference  —  not  an  ideal  situation  for  a  process  that  should  deliver  consistent, 
repeatable  assessments  independent  of  the  individual  conducting  the  assessment. 

The  second  major  dilemma  one  faces  in  assigning  TRLs  is  at  what  level  of  system 
components  to  perform  the  assessment.  There  are  basically  two  approaches  that 
can  be  used:  1)  perform  the  assessment  at  the  system  level,  or  2)  assess  each 
critical  component  individually  and  use  the  individual  TRLs  to  arrive  at  a  system 
TRL.  The  first  approach  applies  the  definitions  to  the  system  as  a  whole  and  is 
probably  the  first  methodology  that  new  users  of  TRLs  consider  using.  While  it 
has  definite  disadvantages,  one  advantage  of  this  approach  is  that  the  standard 
TRL  definitions  are  written  at  the  system  level  and  most  users  are  most  familiar 
with  them  in  this  context.  The  second  approach  attempts  to  assign  a  TRL  to  each 
of  the  critical  components  of  the  system  (which  could  themselves  be  considered 
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technologies)  and  then,  using  these  individual  evaluations,  arrive  at  an  overall 
system  TRL.  This  approach  also  has  several  disadvantages,  but  one  of  its  key 
advantages  is  that  the  decision  maker  is  able  to  tailor  the  evaluation  to  address  the 
technology  components  considered  most  important  to  the  decision  at  hand.  This 
is  especially  useful  for  programs  like  the  TREs  where  there  may  be  many  disparate 
systems  evaluated  against  a  common  set  of  criteria  that  may  relate  to  only  a  subset 
of  the  technologies  present  in  the  candidate  systems. 

To  mitigate  some  of  the  uncertainty  and  ambiguity  inherent  in  this  process,  a 
methodology  is  needed  that  can  resolve  the  ambiguities  in  the  TRL  definitions 
with  consistency,  that  supports  the  goals  of  the  JPEO-CBD  and  JSTO  with 
respect  to  managing  transition  risk  for  emerging  technologies,  and  is  easy  to  apply 
and  understand.  This  methodology  needs  to  combine  the  best  aspects  of  each 
approach  described  above,  while  mitigating  to  the  extent  possible  the 
disadvantages  of  each. 

Individual  versus  Team  TRL  Assessments 

A  key  element  affecting  the  credibility  of  a  TRL  assessment  is  who 
conducts  it.  There  is  nothing  that  says  an  individual  familiar  with  the 
technology  and  the  TRL  process  can’t  be  assigned  to  conduct  the 
evaluation  alone.  While  this  may  be  convenient  in  terms  of  resources 
used,  it  may  not  be  the  best  solution  to  achieve  the  most  robust  and 
credible  assessment. 

Regardless  of  how  objective  and  specific  the  readiness  definitions 
become,  they  remain  subject  to  interpretation.  These  interpretations  will 
always  be  influenced  by  personal  backgrounds,  experiences,  and  biases 
that  will  affect  the  outcome  of  the  evaluation.  One  way  to  mitigate  these 
biases  is  to  have  more  than  one  person  involved  in  the  assessment.  By 
bringing  together  individuals  of  differing  backgrounds  to  evaluate  the 
readiness  of  a  technology,  a  consensus  opinion  can  be  reached  that 
blends  those  biases,  resulting  in  a  stronger  evaluation.  If  the  evaluation 
team  includes  the  principal  stakeholders  for  the  technology,  as  well  as 
representatives  of  the  acquisition  community,  the  credibility  of  the 
assessment  is  reinforced. 


METHODOLOGY 


A  review  of  the  published  information  concerning  TRL  assessments  reveals  little 
in  the  way  of  specific  guidance  that  can  be  directly  applied  to  the  needs  of  the 
JPEO-CBD  with  respect  to  the  conduct  of  TREs.  Several  organizations  and 
individuals  have  published  papers  or  PowerPoint®  presentations  about  their 
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specific  applications,  providing  useful  insights  into  the  process  and  the  potential 
challenges  one  faces  in  assessing  any  technology’s  maturity.  [2,  6,  10,  11,  12]  Their 
usefulness  is  limited,  however,  by  the  very  characteristic  that  prompted  their 
creation  in  the  first  place  —  the  need  to  tailor  and  customize  the  process  to  meet 
the  unique  needs  of  a  specific  technology  area.  This  customization  applies  not 
only  to  the  definitions  themselves,  but  to  their  application  as  well  -  thus,  the  need 
for  a  methodology  that  addresses  the  specific  needs  of  the  CBDP  community. 

The  Technology  Readiness  Assessment  Deskbook  describes  an  assessment 
methodology  that  requires  the  JPM  and  JSTO  to  identify  critical  technologies  for 
evaluation  as  part  of  the  decision  process  for  moving  a  system  on  to  the  next  stage 
of  development.  These  critical  technologies  are  typically  at  the  sub-system  level. 
Each  critical  technology  is  evaluated  separately  for  maturity  and  assigned  a  TRL  — 
no  attempt  is  made  to  assign  an  overall  TRL  to  the  system  at  this  stage.  The 
decision  about  whether  a  system  is  ready  to  move  to  MS  B  or  C  is  made  by  the 
Component  Acquisition  Executive  (with  concurrence  from  the  DUSD(S&T)) 
assessing  all  the  information  provided  in  the  TRA. 

By  contrast,  the  TREs  typically  evaluate  technologies  that  would  represent  the 
sub-systems  or  critical  technologies  in  the  larger  context  of  the  TRA  (see  Example 
1).  To  further  complicate  matters,  the  TRE  technologies  may  themselves  consist 
of  sub-technologies  that  may  be  at  various  levels  of  maturity  (See  Example  2). 

Example  1 

The  purpose  of  TRE-2  was  to  examine  the  readiness  of  trigger 
technologies  for  a  networked,  point  biological  detector.  Where  a  TRA 
for  this  system  would  examine  its  readiness  to  progress  to  MS  B  or  C, 
and  in  the  process  assign  TRLs  to  identified  critical  technologies,  the 
TRE’s  purpose  was  to  identify  candidate  technologies  to  accomplish 
specific  functions  in  the  objective  system.  Thus,  a  critical  technology  for 
the  TRA  becomes  the  evaluation  focus  for  the  TRE. 


Example  2 

There  are  several  sub-systems  of  a  networked,  point  detector  that  could 
be  considered  critical  in  the  context  of  a  TRA.  Besides  the  trigger  sub¬ 
system,  these  could  include:  the  collection  sub-system,  the  network  sub¬ 
system,  the  trigger/ detector  algorithm,  and  command  and  control 
software.  The  candidate  technologies  for  TRE-2  all  consisted  of  various 
combinations  of  these  sub-systems  at  differing  levels  of  maturity. 
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Assigning  TRLs  at  the  sub-system  level  would  seem  to  be  a  good  model  for  the 
TREs;  if  a  methodology  were  available  for  arriving  at  an  overall  decision 
concerning  technology  readiness  similar  to  the  process  used  for  a  TRA  (decision 
maker  reviews  all  available  information  and  makes  a  determination  concerning  the 
system’s  readiness  to  move  forward).  Unfortunately,  the  scope  of  the  TREs 
makes  this  part  of  the  TRA  methodology  unwieldy  for  the  JPEO-CBD  to 
implement. 

An  alternative  approach  would  be  to  assign  TRLs  directly  at  the  system  level.  The 
difficulty  with  applying  the  definitions  at  the  system  level  is  that  the  sub-systems 
can  be  at  various  levels  of  maturity  raising  the  question  as  to  which  definition  best 
represents  the  aggregate  system  maturity. 

Comparing  the  two  approaches,  one  finds  advantages  and  disadvantages  to  each. 
In  the  end,  however,  by  assigning  TRLs  at  the  sub-system  level,  one  is  addressing 
readiness  at  the  functional  level  for  the  system.  This  approach  allows  flexibility  in 
steering  the  assessment  focus  to  the  most  important  technology  components. 

This  is  not  to  say  that  the  sub-system  approach  does  not  present  any  challenges. 
The  dilemma  still  remains  how  to  blend  the  readiness  of  the  various  components 
into  a  single  readiness  level  for  the  system. 

The  TRL  methodology  consists  of  four  basic  steps: 

•  Understand  how  the  technology  will  be  used  (the  program  requirements). 

•  Identify  the  critical  technologies  consistent  with  the  proposed  use. 

•  Assess  the  readiness  of  each  critical  technology  using  the  appropriate 
readiness  variables. 

•  Using  the  results  from  step  3,  determine  the  overall  system  TRL. 

The  readiness  variables  are  fundamental  to  this  process,  but  they  still  don’t 
represent  a  complete  solution.  First,  how  does  one  blend  the  variables  together  to 
arrive  at  an  estimate  of  maturity?  Second,  for  systems  that  may  contain  multiple 
critical  technologies,  how  do  we  use  the  individual  maturity  evaluations  to  estimate 
the  maturity  of  the  system  as  a  whole?  To  begin  to  answer  these  questions  we 
start  by  examining  the  steps  above. 

Identify  the  Critical  Technologies 

Presumably,  a  TRL  evaluation  is  undertaken  to  establish  a  system’s  level  of 
maturity  relative  to  a  specific  purpose.  The  reason  this  is  so  important  is  that  a 
given  technology  may  exhibit  differing  levels  of  maturity  for  different  applications. 
For  example,  if  we  are  interested  in  networked,  point  detectors,  the  critical 
technologies  might  be  the  detection  algorithms,  the  detection  hardware,  and  the 
network  sub-system  (hardware  and  software).  A  candidate  system  may  also 
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include  a  collection  sub-system,  identification  sub-system,  or  any  number  of  other 
“technologies”  that  contribute  to  its  function  but  are  not  considered  critical  for 
the  application  we  are  considering.  By  identifying  the  critical  technologies  with 
respect  to  the  evaluation  purpose,  we  begin  to  prioritize  the  effort  and  shape  the 
evaluation  to  reflect  those  priorities. 

Assess  the  Readiness  of  Each  Critical  Technology 

Having  identified  the  critical  technologies,  the  next  step  is  to  evaluate  their 
maturity  using  the  appropriate  readiness  variables.  This  process  requires  collecting 
detailed  information  about  the  system  such  as:  analytical  studies  that  have  been 
conducted;  testing  that  may  have  been  conducted,  including  who  conducted  it, 
where  it  was  conducted,  the  test  environment,  and  the  results;  the  maturity  of  the 
hardware;  the  level  of  integration  of  the  components;  the  level  of  development  of 
system  software;  the  maturity  of  system  algorithms;  and,  any  other  aspect  of  the 
system  that  may  be  important  for  evaluating  the  readiness  variables.  How  this 
evaluation  is  conducted  is  discussed  in  the  next  section. 

Determine  the  Overall  System  TRL 

Once  the  TRLs  for  the  critical  technologies  have  been  determined,  the  final  step  is 
to  determine  the  overall  system  TRL.  For  a  system  that  consists  of  a  single  critical 
technology,  this  step  is  straightforward  —  the  system  TRL  is  simply  the  TRL  of  the 
critical  technology.  For  a  system  that  consists  of  more  than  one  critical 
technology,  to  the  lowest  TRL  of  a  critical  technology  component  of  the  system  is 
the  overall  system  TRL. 

READINESS  VARIABLES 


Depending  on  the  nature  of  the  technology  involved,  there  are  a  number  of 
variables  of  readiness  that  need  to  be  considered  as  one  evaluates  a  technology: 
Knowledge,  Form  —  Fit  —  Function,  Level  of  Development,  Integration,  Testing, 
and  Environment. 

Knowledge  refers  to  the  level  of  understanding  the  developer  has  about  the 
technology  of  interest  and  its  intended  application.  Levels  of  understanding  range 
from  a  basic  grasp  of  the  underlying  scientific  principles  involved  to  a  complete 
understanding  of  the  operational  environment  and  interfaces  necessary  for  a 
particular  application. 

Form/Fit/Function  refers  to  system  packaging  and  function.  It  measures  how 
close  the  system  is  to  its  final  configuration.  Form/Fit/Function  only  applies  to 
hardware. 
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Level  of  Development  applies  to  algorithms  and  software-based  technologies.  It 
refers  to  how  far  along  the  development  path  the  technology  has  progressed. 

Integration  is  another  readiness  variable  that  applies  to  algorithm  and  software 
technologies.  It  measures  the  achieved  level  of  integration  with  system. 

Testing  refers  to  the  level  or  type  of  testing  that  has  been  performed.  Levels  of 
testing  range  from  exploratory  experiments  and/or  simulation-based  testing  of 
breadboard  systems  to  full-up  operational  testing  of  final  system  configurations. 
The  testing  variable  also  includes  whether  T&E  capabilities,  methods,  models,  and 
tools  exist  for  adequate  operational  testing,  and  how  the  existence  of  the  needed 
infrastructure  affects  the  TRL  supported 

Environment  refers  primarily  to  the  testing  and  operating  environments  the 
technology  has  experienced.  Environments  can  vary  considerably,  but  generally 
range  from  desktop/academic  settings  to  full  operational  missions. 

Table  1  shows  where  each  of  these  variables  is  used  to  estimate  technology 
readiness. 


Table  1.  Readiness  Variable  use  versus  technology  category 


Hardware 

Software 

Alaorithms 

System 

Knowledqe 

X 

X 

X 

Form-Fit-Function 

X 

Level  of  Development 

X 

X 

Intearation 

X 

X 

X 

Testina 

X 

X 

X 

X 

Environment 

X 

X 

X 

X 

Each  of  these  variables  can  be  defined  and  scaled  to  correlate  with  the  standard 
TRL  definitions.  They  can  also  be  tailored  to  the  specific  technology  or 
application  under  consideration.  This  means  we  can  develop  variable  descriptions 
that  are  consistent  with  CBDP  systems  and  address  the  aspects  of  readiness  most 
important  to  an  evaluation  of  these  technologies  (e.g.,  hardware,  software, 
algorithms).  These  descriptions  are  the  foundation  of  the  evaluation  of 
technology  readiness  and  are  based  on  analysis  and  interpretation  of  the  standard 
TRL  definitions. 

Each  variable  description  consists  of  discrete  steps  representing  increasing  levels 
of  maturity  for  that  variable,  as  well  as  the  corresponding  TRL  based  on  the 
standard  definitions.  There  are  three  sets  of  descriptions  for  hardware,  software 
and  algorithms.  The  hardware  descriptions  are  presented  here  and  will  be  used 
throughout  the  remainder  of  the  document  to  explain  and  illustrate  the  evaluation 
methodology. 
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Table  2.  Variable  Descriptions  for  Level  of  Knowledge 


Knowledge 

Highest  TRL 
Supported 

Basic  scientific  principles  observed 

1 

Science  known  to  extent  that  mathematical  and/or  computer 
models  and  simulations  are  possible 

1 

Rigorous  analytical  studies  confirm  basic  principles 

2 

Physical  laboratory  experimental  evidence  confirms  basic 
principles 

3 

Possible  application  exists 

3 

Paper  studies  show  that  application  is  feasible 

3 

Laboratory  experiments  verify  application  is  feasible 

4 

Overall  system  requirements  for  end  user's  application  are 
known 

5 

System  interface  requirements  known 

5 

Operating  environment  for  eventual  system  known 

9 

Table  3.  Variable  Descriptions  for  Level  of  Form/Fit/Function 


Form/Fit/Function 

Highest  TRL 
Supported 

No  system  components,  just  basic  laboratory  research 
equipment  to  verify  physical  principles 

2 

Ad  hoc  and  available  laboratory  components  are  surrogates 
for  system  components 

4 

Some  special  purpose  components  combined  with  available 
laboratory  components 

5 

Components  are  functionally  compatible  with  operational 
system 

6 

Components  are  representative  of  production  components 

7 

Components  are  form,  fit,  and  function  compatible  with 
operational  system 

9 

Table  4.  Variable  Descriptions  for  Testing. 


Testing 

Highest  TRL 
Supported 

None 

2 

Analytical  experiments 

5 

Component  tests 

6 

Developmental  testing 

7 

DT&E  complete 

8 

OT&E  demonstrates  that  system  is  capable  of  performing 
mission  requirements 

9 
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Table  5.  Variable  Descriptions  for  Environment. 


Environment 

Highest  TRL 
Supported 

"Back  of  envelope"  environment 

1 

Desktop  environment 

2 

Academic  environment 

3 

Laboratory  environment,  simulants  only 

4 

Laboratory  environment,  simulants  with  potential  interferants 

6 

Laboratory  environment,  live  agent  with  or  without  interferants 

6 

Outdoor  environment,  simulants  only 

7 

Outdoor  environment,  simulants  with  potential  interferants 

8 

Outdoor  environment,  live  agent  with  or  without  interferants 

9 

Operational  environment 

9 

The  descriptions  in  Tables  2  —  5  should  be  considered  as  representative  of  the 
various  levels  of  maturity  and  not  necessarily  exact  descriptions  to  be  used  for 
every  technology.  In  other  words,  they  should  not  be  considered  a  checklist,  but 
rather  serve  as  guidelines  to  help  estimate  the  maturity  of  a  technology.  They  are, 
however,  assumed  to  build  upon  each  other  to  represent  increasing  levels  of 
readiness.  For  example,  when  evaluating  the  level  of  knowledge  achieved,  the  last 
entry  in  Table  2  assumes  that  all  of  the  steps  above  it  or  their  equivalent  have  been 
accomplished. 

These  tables  illustrate  that  even  at  the  most  basic  level  there  is  still  some  ambiguity 
inherent  in  trying  to  define  levels  of  readiness.  At  each  step  beyond  the  first,  the 
description  can  support  multiple  TRLs  depending  on  the  status  of  other  variables. 
Even  so,  these  descriptions  provide  a  means  for  more  consistent  interpretation  of 
the  TRL  definitions,  as  well  as  an  opportunity  to  tailor  those  definitions  to  a 
particular  technology  area. 

ASSESSING  CRITICAL  TECHNOLOGIES 


Each  critical  technology  is  evaluated  independently  using  the  readiness  variable 
descriptions.  As  each  variable  is  evaluated,  the  result  is  one  or  more  supported 
TRLs  for  each  area.  The  overall  TRL  for  the  technology  is  the  highest  common 
supported  level  across  all  of  the  variables.  Table  6  presents  a  hypothetical  example 
for  a  hardware-based  technology.  The  highest  common  supported  level  is  a  TRL 
of  five,  thus  this  technology  would  be  assessed  to  be  at  TRL  5  with  respect  to 
hardware  development. 
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Table  6.  Individual  technology  assessment  example 


Readiness 

Variable 

Level  Achieved 

TRL  Supported 

Knowledge 

Operating  environment  for 
eventual  system  known 

1,2,  3,  4,  5,  6,  7,  8,9 

Form/Fit/Function 

Components  are 
functionally  compatible  with 
operational  system, 

1,2,  3,  4,  5,6 

Testing 

Component  tests 

1,2,  3,  4,  5,6 

Environment 

Laboratory  environment, 
simulants  with  potential 
interferants 

1,2,  3,  4,  5,  6,7 

If  there  are  no  other  technology  components  of  interest,  the  assessment  may 
conclude  at  this  point  with  the  system  being  assessed  at  TRL  5.  If,  however,  there 
are  other  critical  technologies  or  other  components  associated  with  this 
technology  (detection  algorithms  for  example),  the  process  must  be  repeated  for 
each  component  before  an  overall  TRL  can  be  determined. 

SYSTEM  TRL  ASSESSMENT 


The  overall  system  TRL  is  a  function  of  both  individual  component  readiness,  as 
well  as  the  maturity  of  the  integration  of  those  components.  How  complex  this 
evaluation  becomes,  depends  in  part  on  the  complexity  of  the  system.  There  are 
three  readiness  variables  that  pertain  to  the  system  as  a  whole:  Testing, 
Integration,  and  Environment.  Testing  and  Environment  are  similar  to  what  has 
been  presented  previously  except  now  we  are  looking  at  testing  of  the  integrated 
system.  Tables  7,  8,  and  9  present  the  descriptions  of  the  readiness  variables  for 
this  aspect  of  the  evaluation. 


Table  7.  Variable  descriptions  for  System  Testing 


Testing 

Highest  TRL 
Supported 

None 

2 

Analytical  experiments 

5 

Component  tests 

6 

Developmental  testing 

7 

DT&E  complete 

8 

OT&E  demonstrates  that  system  is  capable  of  performing 
mission  requirements 

9 
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Table  8.  Variable  Descriptions  for  System  Integration 


Integration 

Highest  TRL 
Supported 

No  attempt  at  integration;  still  trying  to  see  whether 
individual  parts  of  the  technology  work 

1 

Paper  studies  indicate  that  system  components  ought  to 
work  together 

1,2 

Laboratory  experiments  with  available  components  show 
that  they  work  together  (lab  kludge) 

3 

Available  components  assembled  into  system  breadboard 

4 

Interfaces  between  components/subsystems  are  realistic 
(Breadboard  with  realistic  interfaces) 

5 

Fidelity  of  system  mock-up  improves  from  breadboard  to 
brassboard 

6 

Laboratory  system  is  high-fidelity  functional  prototype  of 
operational  system 

7 

Prototype  improves  to  pre-production  quality 

8 

System  is  form,  fit,  and  function  design  for  intended 
application  and  weapon  system  platform 

8,9 

System  has  been  installed  and  deployed  in  intended 
weapon  system  platform 

9 

Table  9.  Variable  Descriptions  for  System  Environment 


Environment 

Highest  TRL 
Supported 

"Back  of  envelope"  environment 

1 

Desktop  environment 

2 

Academic  environment 

3 

Laboratory  environment,  simulants  only 

4 

Laboratory  environment,  simulants  with  potential  interferants 

6 

Laboratory  environment,  live  agent  with  or  without  interferants 

6 

Outdoor  environment,  simulants  only 

8 

Outdoor  environment,  simulants  with  potential  interferants 

9 

Outdoor  environment,  live  agent  with  or  without  interferants 

9 

Operational  environment 

9 

The  system  TRL  can  be  no  higher  than  the  lowest  TRL  of  the  component  critical 
technologies.  Thus,  if  a  system  consists  of  three  critical  technologies,  A,  B,  and  C 
at  TRLs  6,  7,  and  9  respectively,  the  system  TRL  can  be  no  higher  than  TRL  6.  It 
could,  however,  be  lower  than  TRL  6,  depending  on  the  level  of  integration, 
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testing,  and  environment  for  the  system  as  a  whole.  Table  10  presents  an  example 
situation  for  the  three-technology  system  described  above.  Even  though  the 
individual  components  are  more  mature,  the  overall  TRL  would  be  TRL  5,  since 
this  is  the  highest  common  TRL  supported  by  the  system  readiness  variables. 


Table  10.  System  Assessment  Example 


Readiness 

Variable 

Level  Achieved 

TRL  Supported 

Integration 

Interfaces  between 
com  ponents/su  bsystems 
are  realistic  (Breadboard 
with  realistic  interfaces) 

1,2,  3, 4,5 

Testing 

Component  tests 

1,2,  3,  4,  5,6 

Environment 

Laboratory  environment, 
simulants  with  potential 
interferants 

1,2,  3,  4,  5,6 

Another  situation  could  arise  where  there  are  multiple  critical  technologies  but  one 
technology  is  “more  critical’5  than  the  others.  This  could  happen,  for  instance, 
when  the  decision  maker  is  particularly  interested  in  a  single  aspect  of  a  system 
such  as  agent  detection,  but  other  aspects  are  also  critical  (the  network  sub-system 
for  instance).  For  example,  using  the  same  three-technology  system,  if  technology 
B  is  the  priority,  the  system  TRL  could  be  as  high  as  TRL  7  if  the  system  readiness 
variables  support  this  level.  The  ceiling  for  the  system  TRL  is  driven  by  the  TRL 
of  the  “most”  critical  technology,  even  if  the  other  “critical”  technologies  are  less 
mature. 

Why  not  just  average  the  sub-system  TRLs? 

Averaging  the  sub-system  TRLs  would  appear  to  be  a  simple, 
straightforward  method  for  determining  a  system  TRL.  At  first,  it  seems 
like  a  simple  way  to  combine  the  TRLs  from  the  various  sub-systems  to 
arrive  at  a  TRL  for  the  system  as  whole.  The  flaw  in  this  logic  is  that  the 
TRL  numbers  do  not  really  represent  a  simple  ordinal  scale,  but  are 
actually  category  labels  representing  maturity  definitions  that  have  specific 
criteria  associated  with  them.  Thus,  it  is  not  necessarily  tme  that  a  system 
that  consists  of  two  sub-systems,  one  at  TRL  6  and  one  at  TRL  8,  would 
have  an  overall  TRL  of  7.  This  would  only  be  tme  if  the  system 
demonstrated  a  level  of  integration,  plus  testing  in  an  appropriate 
environment,  that  justifies  a  TRL  7.  It  would  be  more  accurate  to  say 
that  the  system  could  have  a  TRL  no  higher  than  six  or  eight,  depending 
on  which  sub-system  represents  the  critical  technology.  A  separate 
evaluation  of  maturity  at  the  system  level,  looking  at  integration,  testing 
and  environment  is  necessary  to  determine  whether  the  system  TRL  is  at 
or  below  the  critical  technology  TRL. 
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A  CASE  STUDY 


The  following  example  is  based  in  part  on  TRE  2.  It  provides  some  insight  into 
the  challenges  that  can  present  themselves  in  the  course  of  a  TRL  assessment. 

BACKGROUND 

The  Federal  Business  Opportunities  Vendors  Notice  for  TRE  2  announced  that 
the  US  Army  would  be  sponsoring  a  TRE  for  .  .automatic,  light  weight, 
networked  and  non-networked,  biological  point  detection  technologies. . .”. 
Eventually,  12  systems,  representing  a  wide  variety  of  technologies,  intended 
missions,  and  maturity,  were  evaluated  for  their  potential  to  act  as  triggers  for  a 
system  such  as  the  Joint  Biological  Tactical  Detection  System.  Part  of  the  TRE 
was  assigning  TRLs  to  each  of  the  participating  systems. 

CRITICAL  TECHNOLOGIES 

The  first  step  was  to  identify  the  critical  technologies.  The  starting  point  for  this 
determination  is  the  mission  statement.  From  that  statement  and  additional 
analysis,  the  technology  sub-systems  of  interest  were  determined  to  be: 
detection/identification  sub-systems,  network  sub-systems,  trigger/cue  sub¬ 
systems,  collection  sub-systems,  and  command  and  control  software.  The 
trigger/cue  sub-system  and  the  detection/identification  sub-system  were  expanded 
to  also  include  the  associated  algorithms  for  each. 

Because  there  are  multiple  technology  sub-systems,  the  next  step  was  to  prioritize 
those  technologies  from  most  to  least  important.  In  this  case,  the  trigger/ cue  sub¬ 
system  and  detection/identification  sub-system  were  considered  the  most 
important  (choice  depends  on  the  type  of  system),  and  became  the  critical 
technologies  for  the  TRL  assessment.  Each  of  the  technology  sub-systems  was 
assigned  a  TRL,  but  the  system  TRL  was  based  on  the  readiness  of  the  system  as  a 
whole  plus  the  maturity  of  the  critical  technology. 

SYSTEM  DESCRIPTION 

The  example  system  is  a  biological  trigger  consisting  of  a  trigger/ cue  sub-system 
(including  a  trigger  algorithm),  a  collection  sub-system,  a  network  sub-system,  and 
command  and  control  software.  The  first  challenge  is  collecting  information  of 
sufficient  detail  to  be  able  to  make  assessments  of  maturity  using  the  readiness 
variables.  This  information  can  come  from  many  sources,  with  the  primary  source 
likely  to  be  the  vendor.  Depending  on  the  time  and  other  resources  available,  this 
information  can  be  collected  in  many  ways  including  questionnaires,  interviews, 
phone  conversations,  and  email.  Despite  best  efforts,  the  level  of  completeness 
and  fidelity  of  the  information  is  likely  to  be  far  from  ideal.  The  sidebar  below 
summarizes  the  available  information  for  the  example  system  and  is  representative 
of  the  level  of  completeness  one  can  expect,  especially  for  TREs.  (Although 
based  on  an  actual  participating  system  from  TRE-2,  the  description  has  been 
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edited  to  protect  potential  proprietary  information.  Some  familiarity  with  bio¬ 
detection  technology  is  assumed.) 
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Example  System  Description 


General  Information:  The  system  is  the  result  of  three  years  of 
company-funded  development  and  the  underlying  technology  is  patent 
pending. 

Trigger/ Cue  Sub-System:  The  trigger/ cue  sub-system  technology  is 
based  on  Ultra-Violet  Laser  Induced  Fluorescence.  A  single  laser  diode  is 
used  to  induce  fluorescence  and  elastic  scatter  that  are  measured  using 
three  Photo  Multiplier  Tubes.  Particles  are  interrogated  one  at  a  time 
after  being  drawn  into  the  device  through  an  aerosol  concentrator. 

Trigger  Algorithm:  The  system  uses  two  fluorescence  channels  plus  an 
elastic  scatter  channel  to  size  classify  particles  and  determine  the  presence 
of  biological  material.  Particles  are  classified  into  small,  medium  and 
large  size  bins.  For  each  bin,  an  independent  fluorescence  threshold  is 
established  to  determine  whether  the  particle  has  significant  biological 
content.  This  is  all  accomplished  in  real  time.  The  count  of  particles 
detected  per  second,  as  well  as  the  count  of  biological  particles  per 
second  in  each  size  bin  is  recorded.  A  moving  average  is  computed  from 
the  instantaneous  particle  counts.  An  instantaneous  biological  fraction 
signal  is  derived  for  each  size  bin  by  taking  the  ratio  of  the  biological 
count  rate  to  the  particle  count  rate  for  that  bin.  Also,  a  smoothed 
biological  fraction  signal  is  derived  for  each  size  bin  by  taking  the  ratio  of 
the  moving  average  biological  count  to  the  moving  average  particle  count 
for  each  bin.  A  system  alarm  is  declared  when  the  smoothed  biological 
fraction  exceeds  a  fixed  threshold. 

Collection  Sub-System:  The  system  uses  a  new  collection  sub-system 
that  utilizes  high  flow  axial  vane  fans  as  an  aerosol  impactor.  The  fans 
are  less  than  one  inch  square  and  can  be  placed  in-line  with  the  aerosol 
output  or  can  have  their  own  intake.  When  an  alarm  occurs  the  fans  are 
turned  on  and  collection  is  initiated.  An  independent  laboratory  has 
reported  collection  efficiencies  of  better  than  80%  for  BG  spores. 
Extraction  of  the  sample  is  achieved  by  applying  liquid  to  the  fan 
assembly.  The  fans  are  considered  disposable. 

Detection/Identification  Sub-System:  None. 

Network  Sub-System:  Network  concepts  have  been  developed  and  the 
system  is  physically  capable  of  being  networked,  but  has  not  been 
formally  tested.  Network  detection  algorithms  have  not  been  developed. 
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Command  and  Control  Sub-System:  The  command  and  control 
software  is  mature  and  functional. 

Testing:  The  system  has  begun  testing  for  military,  facility  security,  and 
mail  security  applications  at  representative  locations.  Integration  of  new 
components  of  the  collection  sub-system  has  only  recently  been 
completed  and  the  components  have  not  been  previously  tested  together. 
Trigger  algorithms  continue  to  be  developed  and  tested  in  a  controlled 
laboratory  environment,  as  well  as  field  locations.  Testing  has  only  been 
accomplished  using  BWA  simulants  and  a  limited  number  of  interferants. 


ANALYSIS 

Starting  with  the  least  important  technology  first,  the  command  and  control 
software  is  characterized  as  “. .  .mature  and  functional.”  By  itself  this  would 
appear  to  argue  for  a  fairly  high  TRL  —  TRL  7  to  TRL  9.  Going  through  each  of 
the  software  readiness  variables  we  find  that  the  actual  maturity  may  not  be  quite 
this  high.  The  level  of  Knowledge  supports  up  to  TRL  8;  Function  Development 
supports  TRL  9;  Integration  supports  up  to  TRL  8;  Environment  supports  up  to 
TRL  6;  and  Testing  supports  up  to  TRL  7.  The  command  and  control  software  is, 
therefore  at  TRL  6. 

The  collection  sub-system  is  made  up  of  relatively  mature  components,  but  its 
integration  into  the  example  system  is  relatively  recent  and  has  undergone  only 
limited  laboratory  testing  to  date.  Knowledge  supports  up  to  TRL  9. 
Form/Fit/Function  supports  up  to  TRL  7.  Testing  supports  up  to  TRL  6. 
Environment  also  supports  up  to  TRL  6.  The  collection  sub-system  TRL  is, 
therefore,  TRL  6. 

The  network  sub-system  is  very  immature  and  incomplete  with  respect  to  the 
requirements  of  the  objective  system.  The  level  of  Knowledge  supports  a  TRL  2; 
Form/Fit/Function  supports  up  to  TRL  5;  Testing  supports  up  to  TRL  5;  and 
Environment  supports  up  to  TRL  3.  The  highest  common  TRL  is,  therefore, 

TRL  2  —  beyond  the  idea  stage,  but  still  in  the  concept  development  process. 

The  trigger/ cue  sub-system  is  based  on  UVLIF  technology  and  hardware,  which  is 
fairly  mature  conceptually.  Knowledge  therefore  supports  a  TRL  up  to  TRL  9. 
Form/Fit/Function  of  the  components  is  relatively  far  along  after  three  years  of 
development  and  supports  a  TRL  7.  Testing  of  the  components  is  still  in  the 
developmental  test  stage  and  therefore  supports  TRL  7  as  well.  The  test 
environment  has  been  primarily  laboratory,  with  simulants  and  some  interferants, 
so  the  Environment  would  support  TRL  6.  The  trigger/ cue  sub-system  TRL  is, 
therefore,  TRL  6. 
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The  trigger  algorithms  are  the  real  innovation  of  the  system  and  the  heart  of  the 
technology.  By  definition,  the  algorithms  have  experienced  many  of  the  same 
tests  and  environments  that  the  hardware  has  undergone.  The  difference,  and  the 
reason  that  the  algorithm  maturity  may  be  different  than  the  hardware  used  to 
collect  the  data  they  process,  is  that  the  variables  that  define  their  readiness  are 
different  than  those  of  the  hardware.  In  some  cases  the  descriptions  are  the  same 
as  for  hardware  technologies  (Environment),  in  some  they  address  the  same  issue 
but  use  different  descriptions  (Knowledge  and  Testing),  and  for  others  they  are 
unique  to  algorithms  (Development  and  Integration). 

In  this  case,  the  level  of  Knowledge  about  the  principles  behind  the  algorithms 
supports  up  to  TRL  9;  the  Environment  supports  TRL  6;  the  level  of 
Development  supports  up  to  TRL  9;  Integration  is  at  TRL  8;  and  Testing  supports 
TRL  7.  The  algorithms  are,  therefore,  at  TRL  6. 


Summarizing: 


Technology  Sub-system 

TRL 

T rigger/cue  hardware 

6 

Trigger  algorithm 

6 

Collection 

6 

Network 

2 

Command  &  Control 

6 

The  overall  system  TRL  can  be  no  higher  than  TRL  6,  based  on  the  level  of 
maturity  of  the  critical  technology  (trigger/ cue  plus  trigger  algorithm).  It  can, 
however,  be  less,  depending  on  the  level  of  integration  achieved  and  the  level  of 
system  testing  completed.  All  of  the  recent  testing  has  been  at  the  system  level,  so 
Environment  and  level  of  Testing  are  the  same  as  noted  earlier  and  support  a  TRL 
6.  The  system  is  in  an  advanced  prototype  stage  in  terms  of  form-fit-function  and 
supports  an  Integration  TRL  of  7.  The  overall  system  TRL,  assuming  the 
trigger/ cue  technology  is  the  highest  priority,  is  therefore  TRL  6.  If  the  network 
technology  were  determined  to  be  the  priority,  the  system  would  be  at  TRL  2, 
based  on  the  maturity  of  that  sub-system. 
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Appendix  B  —  TRL,  Definitions 

Tables  B1  and  B2  come  from  the  TRA  Handbook  and  show  the  standard  DAG 
definitions  for  each  TRL,  along  with  additional  supporting  information  intended 
to  make  their  application  easier. 


Table  Bl.  TRL  Definitions,  Descriptions,  and  Supporting  Information. 


TRL 

Definition 

Description 

Supporting  Information 

1 

Basic 
principles 
observed  and 
reported 

Lowest  level  of  technology 
readiness.  Scientific  research 
begins  to  be  translated  into  applied 
research  and  development. 

Examples  might  include  paper 
studies  of  a  technology’s  basic 
properties. 

Published  research  that  identifies  the 
principles  that  underlie  this  technology. 
References  to  who,  where,  when. 

2 

Technology 

concept 

and/or 

application 

formulated 

Invention  begins.  Once  basic 
principles  are  observed,  practical 
applications  can  be  invented. 
Applications  are  speculative,  and 
there  may  be  no  proof  or  detailed 
analysis  to  support  the 
assumptions.  Examples  are  limited 
to  analytic  studies. 

Publications  or  other  references  that  outline 
the  application  being  considered  and  that 
provide  analysis  to  support  the  concept. 

3 

Analytical  and 

experimental 

critical 

function 

and/or 

characteristic 
proof  of 
concept 

Active  research  and  development  is 
initiated.  This  includes  analytical 
studies  and  laboratory  studies  to 
physically  validate  analytical 
predictions  of  separate  elements  of 
the  technology.  Examples  include 
components  that  are  not  yet 
integrated  or  representative. 

Results  of  laboratory  tests  performed  to 
measure  parameters  of  interest  and 
comparison  to  analytical  predictions  for  critical 
subsystems.  References  to  who,  where,  and 
when  these  tests  and  comparisons  were 
performed.  Aspects  of  technology,  including 
predictive  algorithms,  aspects  to  be  simulated 
and  variables  upon  which  predictions  and 
simulations  depend,  have  been  identified. 
Aspects  of  the  technology  are  identified  that 
drive  test  technology  needs. 

4 

Component 
and/or 
breadboard 
validation  in 
[a]  laboratory 
environment 

Basic  technological  components 
are  integrated  to  establish  that  they 
will  work  together.  This  is  relatively 
“low  fidelity”  compared  to  the 
eventual  system.  Examples  include 
integration  of  “ad  hoc”  hardware  in 
the  laboratory. 

System  concepts  that  have  been  considered 
and  results  from  testing  laboratory-scale 
breadboard(s).  References  to  who  did  this 
work  and  when.  Provide  an  estimate  of  how 
breadboard  hardware  and  test  results  differ 
from  the  expected  system  goals.  Data 
available  that  includes  measures  of  physical 
variables  affecting  performance  and  relating 
the  performance  of  components  to  systems,” 
to  the  Supporting  Information  column. 

5 

Component 
and/or 
breadboard 
validation  in 
[a]  relevant 
environment 

Fidelity  of  breadboard  technology 
increases  significantly.  The  basic 
technological  components  are 
integrated  with  reasonably  realistic 
supporting  elements  so  they  can  be 
tested  in  a  relevant  environment. 
Examples  include  “high-fidelity” 
laboratory  integration  of 
components. 

Results  from  testing  a  laboratory  breadboard 
system  that  are  integrated  with  other 
supporting  elements  in  a  simulated  operational 
environment.  How  does  the  “relevant 
environment”  differ  from  the  expected 
operational  environment?  How  do  the  test 
results  compare  with  expectations?  What 
problems,  if  any,  were  encountered?  Was  the 
breadboard  system  refined  to  match  the 
expected  system  goals  more  nearly?  Data 
available  that  includes  measures  of 
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TRL 

Definition 

Description 

Supporting  Information 

performance  that  usually  includes  the  use  of 
live  agents 

6 

System/subsy 
stem  model  or 
prototype 
demonstration 
in  a  relevant 
environment 

Representative  model  or  prototype 
system,  which  is  well  beyond  that  of 
TRL  5,  is  tested  in  a  relevant 
environment.  Represents  a  major 
step  up  in  a  technology’s 
demonstrated  readiness.  Examples 
include  testing  a  prototype  in  a 
high-fidelity  laboratory  environment 
or  in  [a]  simulated  operational 
environment. 

Results  from  laboratory  testing  of  a  prototype 
system  that  is  near  the  desired  configuration  in 
terms  of  performance,  weight,  and  volume. 

How  did  the  test  environment  differ  from  the 
operational  environment?  Who  performed  the 
tests?  How  did  the  test  compare  with 
expectations?  What  problems,  if  any,  were 
encountered?  What  are/were  the  plans, 
options,  or  actions  to  resolve  problems  before 
moving  to  the  next  level?  Technology-specific 
test  technology  and  methodologies  have  been 
developed.  Initial  modeling  and  simulation 
indicate  successful  technology  performance. 

CB  agent  simulants  identified.  Initial  safety 
testing  completed. 

7 

System 
prototype 
demonstration 
in  an 

operational 

environment 

Prototype  near,  or  at,  planned 
operational  system.  Represents  a 
major  step  up  from  TRL  6,  requiring 
demonstration  of  an  actual  system 
prototype  in  an  operational 
environment  such  as  an  aircraft, 
vehicle,  or  space.  Examples  include 
testing  the  prototype  in  a  test  bed 
aircraft. 

Results  from  testing  a  prototype  system  in  an 
operational  environment.  Who  performed  the 
tests?  How  did  the  test  compare  with 
expectations?  What  problems,  if  any,  were 
encountered?  What  are/were  the  plans, 
options,  or  actions  to  resolve  problems  before 
moving  to  the  next  level?  Demonstration  of  a 
model  indicating  affects  of  environmental 
variables  completed.  Safety  data  and  report 
completed.  Performance  validated  using  CB 
simulants. 

8 

Actual  system 
completed  and 
qualified 
through  test 
and 

demonstration 

Technology  has  been  proven  to 
work  in  its  final  form  and  under 
expected  conditions.  In  almost  all 
cases,  this  TRL  represents  the  end 
of  true  system  development. 
Examples  include  developmental 
test  and  evaluation  of  the  system  in 
its  intended  weapon  system  to 
determine  if  it  meets  design 
specifications. 

Results  of  testing  the  system  in  its  final 
configuration  under  the  expected  range  of 
environmental  conditions  in  which  it  will  be 
expected  to  operate.  Assessment  of  whether  it 
will  meet  its  operational  requirements.  What 
problems,  if  any,  were  encountered?  What 
are/were  the  plans,  options,  or  actions  to 
resolve  problems  before  finalizing  the  design? 
Full  technology,  component,  subsystem  or 
system  testing  with  live  agent/simulant. 

Results  match  performance  predictions  from 
simulations.  System  field  testing  using 
simulants. 

9 

Actual  system 

proven 

through 

successful 

mission 

operations 

Actual  application  of  the  technology 
in  its  final  form  and  under  mission 
conditions,  such  as  those 
encountered  in  operational  test  and 
evaluation.  Examples  include  using 
the  system  under  operational 
mission  conditions. 

Operational  test  and  evaluation  reports. 
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Table  B2.  Additional  TRL  Terms  and  Definitions. 


Term 

Definition 

Breadboard 

Integrated  components  that  provide  a  representation  of  a 
system/subsystem  and  that  can  be  used  to  determine  concept 
feasibility  and  to  develop  technical  data.  Typically  configured  for 
laboratory  use  to  demonstrate  the  technical  principles  of 
immediate  interest.  May  resemble  final  system/subsystem  in 
function  only. 

High  Fidelity 

Addresses  form,  fit,  and  function.  [A]  High-fidelity  laboratory 
environment  would  involve  testing  with  equipment  that  can 
simulate  and  validate  all  system  specifications  within  a 
laboratory  setting. 

Low  Fidelity 

A  representative  of  the  component  or  system  that  has  limited 
ability  to  provide  anything  but  first-order  information  about  the 
end  product.  Low-fidelity  assessments  are  used  to  provide  trend 
analysis. 

Model 

A  functional  form  of  a  system,  generally  reduced  in  scale,  near  or 
at  operational  specification.  Models  will  be  sufficiently  hardened 
to  allow  demonstration  of  the  technical  and  operational 
capabilities  required  of  the  final  system. 

Operational  Environment 

Environment  that  addresses  all  the  operational  requirements  and 
specifications  required  of  the  final  system  to  include  platform/ 
packaging. 

Prototype 

A  physical  or  virtual  model  used  to  evaluate  the  technical  or 
manufacturing  feasibility  or  military  utility  of  a  particular 
technology  or  process,  concept,  end  item,  or  system. 

Relevant  Environment 

Testing  environment  that  simulates  the  key  aspects  of  the 
operational  environment. 

Simulated  Operational  Environment 

Either  (1)  a  real  environment  that  can  simulate  all  of  the 
operational  requirements  and  specifications  required  of  the  final 
system  or  (2)  a  simulated  environment  that  allows  for  testing  of  a 
virtual  prototype;  used  in  either  case  to  determine  whether  a 
developmental  system  meets  the  operational  requirements  and 
specifications  of  the  final  system. 
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Appendix  C  —  Software  TRL 
Definitions 


The  US  Army  Communications  Electronics  Command  (CECOM) 
developed  a  set  of  alternative  TRL  definitions  for  software  —based  systems 
that  take  advantage  of  the  DAG  language  that  allows  organizations  to 
augment  or  tailor  the  standard  definitions  to  their  unique  needs.  These 
definitions  were  first  published  outside  CECOM  in  a  report  from  the 
Carnegie  Mellon  Software  Engineering  Institute  in  September  2002  (“Using 
Technology  Readiness  Levels  Scale  to  Support  Technology  Management  in 
the  DoD’s  ATD/STO  Environments,  A  Findings  and  Recommendations 
Report  Conducted  for  Army  CECOM”).  Navy  software  TRL  definitions 
are  almost  identical  to  those  listed  below  and  can  be  found  on  the  Office  of 
Naval  Research  website  at: 


http://www.onr.navy.mil/ files/ auto_ops/ trl_software.asp 


Table  Cl.  Software  TRL  Definitions. 


TRL 

Definition 

Description 

Supporting  Information 

1 

Basic  principles  observed 
and  reported. 

Lowest  level  of  software  technology 
readiness;  a  new  software  domain  is 
being  investigated  by  the  basic 
research  community.  This  level 
extends  to  the  development  of  basic 
use,  basic  properties  of  software 
architecture,  mathematical 
formulations  and  general  algorithms 

Basic  research  activities, 
research  articles,  peer-reviewed 
white  papers,  point  papers,  early 
lab  model  of  basic  concept  may 
be  useful  for  substantiating  the 
TRL  level 

2 

Technology  concept  and/or 
application  formulated. 

Once  basic  principles  are  observed 
practical  applications  can  be 
invented.  Applications  speculative 
and  there  may  be  no  proof  or  detailed 
analysis  to  support  the  assumptions. 
Examples  are  limited  to  analytic 
studies  using  synthetic  data. 

Applied  research  activities 
analytic  studies,  small  code 
units,  papers  comparing 
competing  technologies 

3 

Analytical  and  experimental 
critical  function  and/or 
characteristic  proof  of 
concept 

Active  research  and  development  is 
initiated.  The  level  at  which  scientific 
feasibility  is  demonstrated  through 
analytical  and  laboratory  studies. 

This  level  extends  to  the 
development  of  limited  functionality 
environments  to  validate  critical 
properties  and  analytical  predictions 
using  non-integrated  software 
components  and  partially 
representative  data. 

Algorithms  run  on  a  surrogate 
processor  in  a  laboratory 
environment,  instrumented 
components  operating  in 
laboratory  environment, 
laboratory  results  showing 
validation  of  critical  properties 

4 

Module  and/or  subsystem 
validation  in  a  laboratory 
environment,  i.e.  software 
prototype  development 
environment 

Basic  software  components  are 
integrated  to  establish  that  they  will 
work  together.  They  are  relatively 
primitive  with  regard  to  efficiency  and 
robustness  compared  with  the 
eventual  system.  Architecture 
development  initiated  to  include 
interoperability,  reliability, 
maintainability,  extensibility, 
scalability,  and  security  issues. 

Advanced  Technology 
Development,  Standalone 
prototype  solving  a  synthetic  full- 
scale  problem,  or  standalone 
prototype  processing  fully 
representative  data  sets. 
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TRL 

Definition 

Description 

Supporting  Information 

Emulation  with  current/  legacy 
elements  as  appropriate.  Prototypes 
developed  to  demonstrate  different 
aspects  of  eventual  system. 

5 

Module  and/or  subsystem 
validation  in  a  relevant 
environment 

Level  at  which  software  technology  is 
ready  to  start  integration  with  existing 
systems.  The  Prototype 
implementations  conform  to  target 
environment  /  interfaces. 

Experiments  with  realistic  problems. 
Simulated  interfaces  to  existing 
systems.  System  software 
architecture  established.  Algorithms 
run  on  a  processor(s)  with 
characteristics  expected  in  the 
operational  environment. 

System  architecture  diagram 
around  technology  element  with 
critical  performance 
requirements  defined,  Processor 
selection  analysis,  Sim/Stim 
Laboratory  buildup  plan. 

Software  placed  under 
configuration  management. 
COTS/GOTS  in  the  system 
software  architecture  are 
identified. 

6 

Module  and/or  subsystem 
validation  in  a  relevant  end- 
to-end  environment 

Level  at  which  the  engineering 
feasibility  of  a  software  technology  is 
demonstrated.  This  level  extends  to 
laboratory  prototype  implementations 
on  full-scale  realistic  problems  in 
which  the  software  technology  is 
partially  integrated  with  existing 
hardware/software  systems. 

Results  from  laboratory  testing 
of  a  prototype  package  that  is 
near  the  desired  configuration  in 
terms  of  performance  including 
physical,  logical,  data  and 
security  interfaces. 

Comparisons  to  tested 
environment  to  operational 
environment  analytically 
understood.  Analysis  and  test 
measurements  quantifying 
contribution  to  system-wide 
requirements  such  as 
throughput,  scalability  and 
reliability.  Analysis  of  human- 
computer  (user  environment) 
begun. 

7 

System  prototype 
demonstration  in  an 
operational  high  fidelity 
environment 

Level  at  which  the  program  feasibility 
of  a  software  technology  is 
demonstrated.  This  level  extends  to 
operational  environment  prototype 
implementations  where  critical 
technical  risk  functionality  is  available 
for  demonstration  and  test  in  which 
the  software  technology  is  well 
integrated  with  operational 
hardware/software  systems. 

Critical  technological  properties 
are  measured  against 
requirements  in  a  simulated 
operational  environment 

8 

Actual  system  completed  and 
mission  qualified  through  test 
and  demonstration  in  an 
operational  environment 

Level  at  which  a  software  technology 
is  fully  integrated  with  operational 
hardware  and  software  systems. 
Software  development 
documentation  is  complete.  All 
functionality  tested  in  simulated  and 
operational  scenarios. 

Published  documentation 

Product  technology  refresh  build 
schedule 

Software  resource  reserve 
measured  and  tracked 

9 

Actual  system  proven  through 
successful  mission  proven 
operational  capabilities 

Level  at  which  a  software  technology 
is  readily  repeatable  and  reusable. 

The  software  based  on  the 
technology  is  fully  integrated  with 
operational  hardware/software 
systems.  All  software  documentation 
verified.  Successful  operational 
experience.  Sustaining  software 
engineering  support  in  place.  Actual 
system. 

Production  configuration 
management  reports 

Technology  integrated  into  a 
reuse  “wizard”,  out  year  funding 
established  for  support  activity 
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Appendix  D  —  Algorithm  TRL 
Definitions 


The  definitions  presented  here  were  developed  in  conjunction  with  TRE-2 
and  represent  a  first  attempt  to  develop  measures  of  maturity  for  this 
important  technology  area  in  the  CBDP  arena. 


Table  Dl.  Algorithm  TRL  Definitions. 


TRL 

Definition 

Description 

1 

Basic  principles  observed  and  reported. 

Basic  properties  of  algorithm  defined. 

2 

Technology  concept  and/or  application 
formulated. 

Algorithm  coded.  Experiments  with  synthetic 
data.  Example:  algorithm  tested  with  data 
containing  only  signal,  no  noise. 

3 

Analytical  and  experimental  critical  function 
and/or  characteristic  proof  of  concept. 

Limited  functionality  implementations. 

Experiments  with  small  representative  data  sets. 
Scientific  feasibility  fully  demonstrated.  Example: 
algorithm  fed  more  complicated  (still  synthetic) 
data  representing  intended  target  signals. 

4 

Component  and/or  breadboard  validation  in 
laboratory  environment. 

Experiments  with  full-scale  problems  or  data  sets 
in  a  laboratory  environment.  Example:  algorithm 
presented  with  “real”  data  (simulants  only  at  this 
stage)  obtained  from  hardware  in  a  clean 
environment. 

5 

Component  and/or  breadboard  validation  in 
relevant  environment. 

Algorithm  fully  integrated  with  hardware.  Test  of 
algorithm  in  a  relevant  simulated  environment 
using  “live”  data.  Begin  development  of  signature 
database.  Example:  algorithm  presented  with 
“real”  data  (primarily  simulants,  but  may  include 
actual  targets)  in  an  environment  that  includes 
possible  interferants.  May  be  conducted  in  a 
controlled  environment  such  as  an  aerosol 
chamber  or  breeze  tunnel. 

6 

System/subsystem  model  or  prototype 
demonstration  in  a  relevant  environment. 

Algorithm  tested  in  a  representative  model  or 
prototype  system,  which  is  well  beyond  that  of 

TRL  5.  Signature  database  substantially 
complete  for  initial  targets.  Example:  algorithm 
presented  with  more  complex  situations  such  as 
multiple  targets  and  interferants.  May  be 
conducted  in  a  controlled  environment  such  as  an 
aerosol  chamber  or  breeze  tunnel. 

7 

System  prototype  demonstration  in  an 
operational  environment. 

Algorithm  demonstrated  in  an  operational 
environment.  Represents  a  major  step  up  from 

TRL  6,  requiring  demonstration  of  an  actual 
system  prototype  in  an  operational  environment. 
Example:  algorithm  demonstrated  in  a  system 
prototype  in  an  operational  environment  that 
includes  all  initial  targets  and  expected 
interferants. 

8 

Actual  system  completed  and  qualified 
through  test  and  demonstration. 

Algorithm  has  been  proven  to  work  in  its  final 
form  and  under  expected  conditions.  Signature 
database  for  initial  targets  (not  simulants) 
complete.  In  almost  all  cases,  this  TRL 
represents  the  end  of  true  algorithm 
development.. 

9 

Actual  system  proven  through  successful 
mission  operations. 

Actual  application  of  the  algorithm  in  its  final  form 
and  under  mission  conditions,  such  as  those 
encountered  in  operational  test  and  evaluation. 
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Appendix  E  —  Medical  TRL 
Definitions 


Medical-related  items  require  TRL  definitions  and  descriptions  that  are 
appropriate  to  the  technologies  upon  which  they  are  based  and  that  account 
for  the  statutes  and  regulations  that  govern  their  development  and  use.  In 
recognition  of  these  factors,  the  U.S.  Army  Medical  Research  and  Material 
Command  (USAMRMC)  took  the  initiative  to  establish  appropriate 
definitions,  descriptions,  and  processes  in  the  context  of  military  medical 
research  and  development  and  the  statutory  and  regulatory  requirements 
under  the  stewardship  of  the  FDA. 

USAMRMC’s  TRL  definitions  for  medical  equipment  and  pharmaceuticals 
represent  one  of  the  most  comprehensive  efforts  to  date  to  customize  the 
standard  TRL  definitions  to  a  particular  technology  area.  These  efforts  are 
indicative  of  what  is  required  to  make  the  definitions  relevant  for 
technologies  that  have  unique  requirements  and  milestones  that  must  be 
met  before  they  can  be  considered  “mature”  enough  for  fielding  or 
transition. 


Table  El.  Medical  TRL  Definitions. 


TRL 

DoD  Description 
(DAG,  Oct  2004) 

Medical  Description 
(Oct  2004) 

1 .  Basic  principles  observed  and 
reported. 

Lowest  level  of  technology  readiness.  Scientific 
research  begins  to  be  translated  into  applied 
research  and  development.  Examples  might 
include  paper  studies  of  a  technology’s  basic 
properties. 

Earliest  level  of  technology 
readiness.  Active  monitoring  of 
scientific  knowledge  base. 

Scientific  findings  are  reviewed  and 
assessed  as  a  foundation  for 
characterizing  new  technologies 

2.  Technology  concept  and/or 
application  formulated. 

Invention  begins.  Once  basic  principles  are 
observed,  practical  applications  can  be  invented. 
Applications  are  speculative  and  there  may  be  no 
proof  or  detailed  analysis  to  support  the 
assumptions.  Examples  are  limited  to  analytic 
studies. 

Focus  efforts  on  practical 
applications  based  on  basic 
principles  observed.  Generation  of 
scientific  “paper  studies”  that  review 
and  generate  research  ideas, 
hypothesis,  and  experimental 
designs  for  addressing  the  related 
scientific  issues. 
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TRL 

DoD  Description 
(DAG,  Oct  2004) 

Medical  Description 
(Oct  2004) 

3.  Analytical  and  experimental  critical 
function  and/or  characteristic  proof  of 
concept. 

Active  research  and  development  is  initiated. 

This  includes  analytical  studies  and  laboratory 
studies  to  physically  validate  analytical  predictions 
of  separate  elements  of  the  technology. 

Examples  include  components  that  are  not  yet 
integrated  or  representative. 

Research,  data  collection,  and 
analysis  begin  in  order  to:  test 
hypothesis;  explore  alternative 
concepts;  identify  and  evaluate 
critical  technologies  and 
components;  and  research  and 
eventual  development  of  candidate 
countermeasure(s).  Conduct  non- 
clinical  studies  to  support  models 
based  on  presumed  battlefield 
conditions. 

4.  Component  and/or  breadboard 
validation1  in  laboratory  environment. 

Basic  technological  components  are  integrated  to 
establish  that  they  will  work  together.  This  is 
relatively  “low  fidelity”  compared  to  the  eventual 
system.  Examples  include  integration  of  “ad  hoc” 
hardware  in  the  laboratory. 

Laboratory  research  to  refine 
hypothesis  and  identify  relevant 
parametric  data  required  for 
technological  assessment  in  a 
rigorous  experimental  design. 
Exploratory  study  of  critical 
technologies  for  effective  integration 
into  candidate(s). 

Assess  safety  and  efficacy  utilizing 
animal  model(s). 

Propose  assays,  surrogate  markers, 
and  endpoints  to  be  used  during 
non-clinical  and  clinical  studies  to 
evaluate  and  characterize 
candidate(s). 

5.  Component  and/or  breadboard 
validation2  in  relevant  environment. 

Fidelity  of  breadboard  technology  increases 
significantly.  The  basic  technological  components 
are  integrated  with  reasonably  realistic  supporting 
elements  so  it  can  be  tested  in  a  simulated 
environment.  Examples  include  “high  fidelity” 
laboratory  integration  of  components. 

Conduct  non-clinical  research 
studies  involving  data  collection  and 
analysis  in  well-defined  systems 
with  highly  characterized  lots  of 
candidate(s)  produced  and  further 
development  of  selected 
candidates. 

Develop  a  robust  and  reproducible 
manufacturing  process  amenable  to 
current  good  manufacturing  practice 
(cGMP). 

Qualify  assays  for  potency,  purity, 
identity  and  quality. 

Qualify  surrogate  markers  for 
efficacy  in  animal  models 

6.  System/subsystem  model  or 
prototype  demonstration  in  a  relevant 
environment. 

Representative  model  or  prototype  system,  which 
is  well  beyond  that  of  TRL  5,  is  tested  in  a 
relevant  environment.  Represents  a  major  step 
up  in  a  technology’s  demonstrated  readiness. 
Examples  include  testing  a  prototype  in  a  high- 
fidelity  laboratory  environment  or  in  simulated 
operational  environment. 

Manufacture,  release  and  stability 
test  good  manufacturing  practice 
(GMP)  pilot  lots 

Conduct  Good  Laboratory  Practice 
(GLP)  safety  studies 

Prepare  and  Submit  IND 

Conduct  Phase  1  clinical  trial 

1  Not  “validation”  as  defined  by  FDA.  FDA-type  validations  will  be  done  at  TRL  6-8  and  are  needed 
for  licensure. 

2  Not  “validation”  as  defined  by  FDA.  FDA-type  validations  will  be  done  at  TRL  6-8  and  are  needed 
for  licensure. 
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TRL 

DoD  Description 
(DAG,  Oct  2004) 

Medical  Description 
(Oct  2004) 

7.  System  prototype  demonstration  in 
an  operational  environment. 

Prototype  near,  or  at,  planned  operational  system. 
Represents  a  major  step  up  from  TRL  6,  requiring 
demonstration  of  an  actual  system  prototype  in  an 
operational  environment  such  as  an  aircraft, 
vehicle,  or  space.  Examples  include  testing  the 
prototype  in  a  test  bed  aircraft. 

Conduct  Phase  2  clinical  trial. 
Establish  final  dose,  dose  range, 
schedule,  and  route  of 
administration. 

Data  collected,  presented,  and 
discussed  with  FDA  at  meeting 
(Type  B). 

Clinical  endpoints  and  supporting 
animal  test  plans  agreed  to  by  FDA. 
Complete  process  validation  and 
initiate  consistency  lot  production. 

8.  Actual  system  completed  and 
qualified  through  test  and 
demonstration. 

Technology  has  been  proven  to  work  in  its  final 
form  and  under  expected  conditions.  In  almost  all 
cases,  this  TRL  represents  the  end  of  true  system 
development.  Examples  include  developmental 
test  and  evaluation  of  the  system  in  its  intended 
weapon  system  to  determine  if  it  meets  design 
specifications. 

Complete  production  &  testing  of 
consistency  lots. 

Conduct  Phase  3  clinical  trials,  if 
applicable. 

Submit  BLA/NDA  to  FDA 

Obtain  FDA  approval. 

9.  Actual  system  proven  through 
successful  mission  operations. 

Actual  application  of  the  technology  in  its  final 
form  and  under  mission  conditions,  such  as  those 
encountered  in  operational  test  and  evaluation. 
Examples  include  using  the  system  under 
operational  mission  conditions. 

Post  licensure/approval  use  of 
product. 

Fulfill  post-licensure  commitments, 
if  required. 
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Appendix  F  —  Readiness 
Variables 


HARDWARE: 


Knowledge  (Hardware) 

Highest  TRL 
Supported 

Basic  scientific  principles  observed 

1 

Rigorous  analytical  studies  confirm  basic  principles 

2 

Physical  laboratory  experimental  evidence  confirms 
basic  principles;  Possible  application  exists 

3 

Laboratory  experiments  verify  application  is  feasible 

4 

Overall  system  requirements  for  end  user's  application 
are  known 

5 

System  interface  requirements  known 

7 

Operating  environment  for  eventual  system  known 

9 

Form-Fit-Function  (Hardware) 

Highest  TRL 
Supported 

No  system  components,  just  basic  laboratory  research 
equipment  to  verify  physical  principles 

2 

Ad  hoc  and  available  laboratory  components  are 
surrogates  for  system  components 

4 

Some  special  purpose  components  combined  with 
available  laboratory  components 

5 

Components  are  functionally  compatible  with 
operational  system 

6 

Components  are  representative  of  production 
components 

7 

Components  are  form,  fit,  and  function  compatible  with 
operational  system 

9 

Testing  (Hardware) 

Highest  TRL 
Supported 

None 

2 

Analytical  experiments 

5 

Component  tests 

6 

Developmental  and  environmental  testing 

7 

DT&E  demonstrates  that  the  system  is  capable  of 
meeting  mission  requirements 

8 

OT&E  demonstrates  that  system  is  capable  of 
performing  mission  requirements 

9 

Environment  (Hardware) 

Highest  TRL 
Supported 

"Back  of  envelope"  environment 

1 

Academic  environment 

3 
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Laboratory  environment,  simulants  only 

4 

Laboratory  environment,  simulants  with  potential 
interferants 

5 

Laboratory  environment,  live  or  inactivated  agent  with  or 
without  interferants 

6 

Outdoor  environment,  simulants  only 

6 

Outdoor  environment,  simulants  with  potential 
interferants 

7 

Outdoor  environment,  live  or  inactivated  agent  with  or 
without  interferants 

8 

Operational  environment 

9 

SOFTWARE: 


Knowledge  (Software) 

Highest  TRL 
Supported 

Know  what  software  needs  to  do  in  general  terms 

2 

Have  some  concept  in  mind  that  may  be  realizable  in 
software 

2 

Have  an  idea  that  captures  the  basic  principles  of  a 
possible  algorithm 

2 

Initial  analysis  shows  what  major  functions  need  to  be 
done  in  software 

2 

Initial  analysis  gives  some  idea  of  what  software 
architecture  will  look  like 

2 

Analysis  provides  detailed  knowledge  of  specific 
functions  software  needs  to  perform 

2 

Know  what  hardware  software  will  be  hosted  on 

2 

Know  what  output  devices  are  available 

3 

Outline  of  software  algorithms  available 

3 

Know  what  software  is  presently  available  that  does 
similar  task  (Inventory  completed) 

3 

Know  limitations  of  presently  available  software 
(Analysis  of  current  software  completed) 

4 

Analysis  of  data  requirements  and  formats  completed 

4 

Analysis  of  internal  interface  requirements  completed 

5 

External  interfaces  described  as  to  source,  format, 
structure,  content,  and  method  of  support 

5 

Inventory  of  external  interfaces  completed 

7 

Analysis  of  timing  constraints  completed 

7 

Analysis  of  database  structures  and  interfaces 
completed 

9 
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Development  (Software) 

Highest  TRL 
Supported 

None 

2 

Software  architecture  defined  in  terms  of  major 
functions  to  be  performed 

2 

Preliminary  algorithm  development  completed 

3 

Software  programming  language  selected 

3 

Formal  software  test/inspection  protocol  defined 

3 

Algorithms  converted  to  pseudocode 

4 

Requirements  for  each  function  established 

4 

Coding  of  individual  functions/modules  completed 

9 

Integration  (Software) 

Highest  TRL 
Supported 

None 

2 

Existing  software  examined  for  possible  reuse 

4 

Functions  integrated  into  modules 

5 

"Alpha"  version  software  has  been  released 

6 

"Beta"  version  software  has  been  released 

9 

Testing  (Software) 

Highest  TRL 
Supported 

None 

2 

Metrics  established 

3 

Designs  verified  through  formal  inspection  process 

4 

Individual  functions  tested  to  verify  that  they  work 

5 

Individual  modules  and  functions  tested  for  bugs 

5 

Individual  modules  tested  to  verify  that  the  module 
components  (functions)  work  together 

6 

Verification,  Validation  and  Accreditation  (VV&A) 
initiated 

6 

Each  software/system  interface  tested  individually  under 
stressed  and  anomalous  conditions 

6 

VV&A  in  process  with  the  verification  step  that  software 
specifications  are  met  completed 

7 

VV&A  validation  step  completed,  software  works  in  real 
world 

7 

VV&A  accreditation  step  completed,  software 
authorized  for  use  in  intended  system 

7 

DT&E  completed,  software  meets  specifications 

8 

OT&E  completed,  software  system  is  operational 

9 
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Environment  (Software) 

Highest  TRL 
Supported 

"Back-of-the-envelope"  concept 

1 

Academic  environment 

3 

Individual  functions  or  modules  demonstrated  in  a 
laboratory  environment 

4 

Integration  of  modules/functions  demonstrated  in  a 
laboratory  environment 

5 

Representative  software  system  or  prototype 
demonstrated  in  a  laboratory  environment 

6 

Fully  integrated  software  prototype  demonstrated  in 
actual  or  simulated  operational  environment 

7 

Software  qualified  through  test  and  evaluation  in  actual 
system  (DT&E  completed) 

8 

Actual  mission  software  "flight  proven"  through 
successful  mission  operations  (OT&E  completed) 

9 

ALGORITHMS: 


Knowledge  (Algorithms) 

Highest  TRL 
Supported 

Basic  scientific  principles  observed 

1 

Rigorous  analytical  studies  confirm  basic  principles 

2 

Analysis  provides  detailed  knowledge  of  specific 
functions  algorithm  needs  to  perform 

4 

Implementation  software  identified 

5 

Internal  and  external  interfaces  identified 

6 

Experimental  evidence  confirms  basic  principles 

9 

Development  (Algorithms) 

Highest  TRL 
Supported 

None 

1 

Algorithm  logic  "sketched  out" 

2 

Preliminary  algorithm  development  started 

3 

Software  programming  language  selected 

4 

Algorithm  development  complete 

6 

Algorithm  converted  to  pseudocode 

7 

Algorithm  coding  complete 

9 

Integration  (Algorithms) 

Highest  TRL 
Supported 

None 

2 

Stand  alone 

4 

Working  version  available  on  developmental  platform 

5 

Algorithm  integrated  with  prototype  system 

7 

Algorithm  integrated  with  operational  system 

9 
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Environment  (Algorithms) 

Highest  TRL 
Supported 

"Back  of  envelope"  environment 

1 

Academic  environment 

3 

Laboratory  environment,  simulants  only 

4 

Laboratory  environment,  simulants  with  potential 
interferants 

5 

Laboratory  environment,  live  or  inactivated  agent  with  or 
without  interferants 

6 

Outdoor  environment,  simulants  only 

6 

Outdoor  environment,  simulants  with  potential 
interferants 

7 

Outdoor  environment,  live  or  inactivated  agent  with  or 
without  interferants 

8 

Operational  environment 

9 

Testing  (Algorithms) 

Highest  TRL 
Supported 

None 

2 

Experiments  with  synthetic  data  to  verify  basic 
functionality 

3 

Experiments  with  small  scale  operationally 
representative  data  sets  (synthetic) 

4 

Experiments  with  full  scale  data  sets  (synthetic) 

5 

Initial  experiments  with  limited-scale  "live"  data 

6 

Experiments  with  full  scale  live  data  sets 

7 

DT&E  demonstrates  that  system  meets  procurement 
specifications 

8 

OT&E  demonstrates  that  system  is  capable  of 
performing  mission  requirements 

9 
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SYSTEM: 


Integration 

Highest  TRL 
Supported 

No  attempt  at  integration;  still  trying  to  see  whether 
individual  parts  of  the  technology  work 

1 

Paper  studies  indicate  that  system  components  ought  to 
work  together 

2 

Laboratory  experiments  with  available  components 
show  that  they  work  together  (lab  kludge) 

3 

Available  components  assembled  into  system 
breadboard 

4 

Interfaces  between  components/subsystems  are 
realistic  (Breadboard  with  realistic  interfaces) 

5 

Fidelity  of  system  mock-up  improves  from  breadboard 
to  low-fidelity  prototype 

6 

Laboratory  system  is  high-fidelity  functional  prototype  of 
operational  system 

7 

Prototype  improves  to  pre-production  quality 

8 

System  is  form,  fit,  and  function  design  for  intended 
application  and  weapon  system  platform 

9 

System  has  been  installed  and  deployed  in  intended 
weapon  system  platform 

9 

System  Environment 

Highest  TRL 
Supported 

"Back  of  envelope"  environment 

1 

Academic  environment 

3 

Laboratory  environment,  simulants  only 

4 

Laboratory  environment,  simulants  with  potential 
interferants 

5 

Laboratory  environment,  live  or  inactivated  agent  with  or 
without  interferants 

6 

Outdoor  environment,  simulants  only 

6 

Outdoor  environment,  simulants  with  potential 
interferants 

7 

Outdoor  environment,  live  or  inactivated  agent  with  or 
without  interferants 

8 

Operational  environment 

9 

Testing 

Highest  TRL 
Supported 

None 

2 

Analytical  experiments 

5 

Component  tests 

6 

Developmental  and  environmental  testing 

7 

DT&E  demonstrates  that  the  system  is  capable  of 
meeting  mission  requirements 

8 

OT&E  demonstrates  that  system  is  capable  of 
performing  mission  requirements 

9 
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Appendix  G  -  Manufacturing 
Readiness  Levels 


Table  Gl.  Manufacturing  Readiness  Level  Definitions. 


T 

R 

L 

M 

R 

L 

MRL 

Definition 

Description 

Key  Manufacturing  Issues 

Acquisition  Phase 

1 

1 

NA 

2 

2 

NA 

3 

3 

Manufacturing 

concepts 

identified 

Identification  of 
current 

manufacturing 
concepts  or 
producibility  needs 
based  on  laboratory 
studies. 

-Have  critical  manufacturing 
processes  been  identified? 

-What  are  the  initial  assumptions 
and  understanding  regarding 
availability  of  manufacturing 
capabilities? 

-  Is  there  a  similar  manufacturing 
process  in  use?  If  so,  what  can 
be  learned  from  the  process? 

-Will  a  manufacturing  process  or 
processes  have  to  be  developed 
or  can  an  existing  process  be 
modified? 

Pre-concept 

refinement 

4 

4 

Laboratory 

manufacturing 

processes 

identified 

Key  processes 
identified  and 
assessed  in 
laboratory.  Risk 
mitigation  strategies 
identified  to  address 
manufacturing/produ 
cibility  shortfalls. 
Preliminary  Cost  as 
an  Independent 
Variable  (CAIV) 
targets  set  and  cost 
drivers  identified. 

-What  are  the  key  properties  of 
the  end  item  that  are  critical  to 
maintain  expected 
functionality/performance 
(materials,  signal,  orientation, 
interface  characteristics,  noise 
levels  etc)? 

-What  is  the  range  of  tolerances 
for  the  key  properties  and 
subsystems  to  retain 
functionality/performance? 

-  How  is  this  information  being 
tracked/documented? 

-Have  subcontract  components 
been  included  as  part  of  the 
analysis? 

-Do  available  manufacturing 
capabilities  support  the  key 
performance  parameters?  Has 
this  been  tested  or  is  it  assumed? 
-Has  a  risk  management  strategy 
been  identified  to  address 
manufacturing  /producibility  of 
critical  manufacturing  processes? 
-Have  preliminary  Cost  as  an 
Independent  Variable  (CAIV) 
targets  been  set  and  cost  drivers 
identified? 

Concept  refinement 
leading  to  a  Milestone 

A  decision 

5 

5 

Manufacturing 

process 

development 

Trade  studies  and 
laboratory 

experiments  result  in 
development  of  key 
manufacturing 
processes  and  initial 
sigma  levels  needed 
to  satisfy  CAIV 
targets.  Preliminary 
manufacturing 
assembly  sequences 
identified.  Process, 
tooling,  inspection, 
and  test  equipment 
in  development. 
Significant 

-Have  key  manufacturing  process 
steps  been  outlined? 

-Has  it  been  demonstrated  that 
critical  parameters  can  be 
measured  or  controlled  to  the 
required  level? 

-Is  there  an  initial  manufacturing 
plan  on  sources  of  key 
components? 

-Can  a  cost  be  associated  with 
each  critical  component  with 
reasonable  confidence  or  will 
additional 

information/development  be 
required?  If  not,  what  is  the  plan 
to  acquire  the  information? 
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T 

R 

L 

M 

R 

L 

MRL 

Definition 

Description 

Key  Manufacturing  Issues 

Acquisition  Phase 

engineering  and 
design  changes. 
Quality  and  reliability 
levels  not  yet 
established.  Tooling 
and  machines 
demonstrated  in  the 
laboratory.  Physical 
and  functional 
interfaces  have  not 
been  completely 
defined. 

-Do  initial  cost  estimates  show 
the  need  for  potential 
manufacturing  process  tradeoffs? 

6 

6 

Critical 

manufacturing 

Processes 

demonstrated 

Critical 

manufacturing 
processes  initially 
demonstrated  for  the 
relevant  environment 
(laboratory  or 
simulated 
operational 
environment).  Initial 
goals  established  for 
yields.  Process  and 
tooling  generally 
mature.  Frequent 
design  changes  still 
occur.  Investments  in 
machining  and 
tooling  identified. 
Quality  and  reliability 
levels  identified. 

Design  to  cost  goals 
identified. 

-Have  the  critical  manufacturing 
processes  been  shown  to 
produce  a  product  of  acceptable 
performance 

-Has  the  selected  manufacturing 
process  demonstrated  the  same 
level  of  performance  over 
multiple  (dozens,  hundreds, 
thousands)  production  items? 

-Is  the  process  well  documented? 
-What  tooling  still  needs  to  be 
developed?  What  is  the  level  of 
risk  in  developing  this  tooling? 
-What  analysis  was  used  to 
develop  predicted  quality 
levels/yields?  Is  this  an 
acceptable  risk  or  is  further 
production  testing  required? 

-How  were  manufacturing  costs 
identified  and  what  were  the 
assumptions?  Do  they  require 
additional  development  and 
refinement  of  manufacturing 
processes  to  meet  goals?  Are 
they  based  on  technology 
breakthroughs? 

Technology 
Development  (TD) 
leading  to  a 

Milestone  B 
decision 

7 

7 

Prototype 

manufacturing 

system 

Prototype  system 
built  based  on 
mature  tooling.  Initial 
sigma  levels 
established,  based 
on  yields  and  quality 
data  from  laboratory 
or  simulations. 

Design  changes 
decrease 

significantly.  Process 
tooling  and 
inspection  and  test 
equipment 
demonstrated  in  pre- 
production 
environment. 
Manufacturing 
processes  well 
understood.  CAIV 
and  design  to  cost 
goals  validated. 

-For  quality  levels,  has  the 
characteristics  of  an  acceptable 
product/component  been  clearly 
defined  for  initial  production 
(critical  characteristics  and 
limits)? 

-What  process  inputs  affect 
product  quality  and  how  are  they 
being  controlled?  Has  full-scale 
production  equipment  been 
proven  to  produce  an  acceptable 
component  within  targeted  quality 
levels? 

-Have  the  CAIV  and  design  to 
cost  goals  been  validated. 

System  Development 
and 

Demonstration  (SDD) 

8 

8 

Manufacturing 
process  maturity 
demonstration 

Manufacturing 

processes 

demonstrate 

SDD,  leading  to  a 
Milestone  C 
decision  and  LRIP 
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T 

R 

L 

M 

R 

L 

MRL 

Definition 

Description 

Key  Manufacturing  Issues 

Acquisition  Phase 

acceptable  yield  and 
producibility  levels 
for  pilot  line,  low  rate 
initial  production 
(LRIP),  or  similar 
item  production.  All 
design  requirements 
satisfied. 

Manufacturing 
processes  well 
understood  and 
controlled  to  4-sigma 
or  appropriate  quality 
level.  Minimal 
investment  in 
machine  and  tooling 
(should  have 
completed 
demonstration  in  at 
least  a  low  rate 
production 
environment).  Cost 
estimates  less  than 

125  percent  of  cost 
goals  (e.g.,  design- 
to-cost  and  CAIV 
goals  met  for  LRIP). 

9 

9 

Manufacturing 

processes 

proven 

Manufacturing  line 
operating  at  desired 
sigma  or  similar 
quality  level.  Stable 
design  and 
production.  All 
manufacturing 
processes  controlled 
to  6-sigma  or 
appropriate  quality 
level.  Cost  estimates 
less  than  110 
percent  of  cost  goals 
or  meets  cost  goals 
(e.g.,  CAIV  and 
design-to-cost  goals 
met). 

Production, 
deployment,  and 
support 
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Appendix  H  -  Technology 
Transition  Agreement  Format 

TECHNOLOGY  TRANSITION  AGREEMENT 

OF 

(INSERT  S&T  TECHNOLOGY  NAME  AND 
CONTROL#  HERE) 

TO 

(INSERT  ACQUISITION  PROGRAM  OF 
RECORD  NAME  HERE) 


1.  Description  of  Technology  or  Capability  to  be  Delivered 

Responsible  party:  JSTO.  Briefly  describe  what  the  S&T  activity  intends  to 
develop  for  transition  to  the  acquisition  program.  Include  capability  delivery 
dates 

2.  Target  Acquisition  Program 

Responsible  party:  JPEO-CBD.  Provide  a  brief  description  of  the 
acquisition  program  to  receive  the  technology/ product.  Include: 

a.  Major  program  objectives. 

b.  Current  phase  of  the  acquisition  life  cycle. 

c.  Projected  initial  operational  capability  date. 

3.  Acquisition  Program  Technology  Need 

Responsible  party:  JPEO-CBD.  Identify  the  technology  needs  of  the 
acquisition  program  that  S&T  is  expected  to  provide.  Briefly  describe  the 
benefit  that  the  technology/ product  will  bring  to  the  acquisition  program: 

a.  Relate  the  benefit  to  the  I  CD,  CDD,  CPD,  etc. 

b.  Include  need  dates  for  specific  capabilities. 

4.  Integration  Strategy 

Responsible  party:  JPEO-CBD.  Describe  the  process  for  integrating  the 
technology /product  into  the  acquisition  program.  Include  the  following 
elements  of  acquisition  strategy: 

a.  Evolutionary  acquisition,  block  upgrade,  etc. 

b.  Required  contractor-to-contractor  agreements 

c.  Acquisition  Program  Element  (PE)  numbers  funding  the 
transition 

d.  Annual  PE  funding  levels  committed  to  the  transition  program 

e.  Transition  Fiscal  Year  (FY) 

f.  Statement  conveying  the  level  of  commitment.  For  example: 
Intent:  “Upon  successful  demonstration  of  key  performance 
requirements  (exit  criteria),  JPM  XXX  (acquisition  program 
office)  may  integrate  XXX  (product  performer  is  delivering) 
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into  XXX  (acquisition  program  that  will  integrate  deliverable) 
commencing  in  FYXX  (transition  year)  under  PE  XXXXXXX 
Project  XXXX  (FYDP  budget  profile) 

5.  Points  of  Contact 

Identify  personnel  responsible  for  acquisition  project  management,  S&T 
Technology  Manager,  and  financial  point  of  contact  (POC): 

a.  Project  Manager  POC  information 

b.  Technology  Manager  (Principal  Investigator)  POC  information 

c.  Financial  POC  information 

d.  ICT  manager  POC  information 

e.  T&E  POC  information 

6.  Requirements 

Responsible  party:  JPEO-CBD  in  coordination  with  JRO-CBRND. 

Identify  the  governing  source  of  the  capability  requirement:  the  ICD,  CDD, 
or  other  official  reference  documenting  the  capability  need  and  date 
approved. 

7.  Test  and  Evaluation  Strategy  (See  also  the  Test  and 
Evaluation  Strategy  section  on  p.18) 

Responsible  party:  JSTO  in  coordination  with  JPEO-CBD,  Joint  T&E 
Executive 

a.  Test  Methodology  development  to  be  accomplished. 

b.  Concept  of  employment 

c.  Early  operational  assessment  opportunity 

d.  Key  performance  parameters 

e.  Test  Infrastructure  to  be  provided 

f.  T&E  POC  and  contact  information 

8.  Technical  Details  and  Programmatics 

Responsible  party:  JSTO 

a.  Technology  —  Current  Status 

Status  Summary.  Summarize  current  state  of  development. 
Identify: 

i.  Primary  areas  where  additional  development  is  required. 

ii.  Estimate  of  current  TRL. 

b.  Risk  Analysis.  Prioritize  and  discuss  major  areas  of  technical 
risk.  Identify  planned  mitigation  activities  to  address  technical 
risk  (e.g,  producibility,  affordability,  sustainability). 
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Top  Risks 

Brief  Descriptions 

Mitigation  Strategv 

c.  Technology  Development  Strategy  (See  also  the  TPS 
section  for  content,  page  17) 

Outline  planned  approach.  Include: 

i.  Efforts  required  beyond  those  currently  underway. 

ii.  Integration  plans  if  multiple  projects  are  planned. 

iii.  Form,  fit,  and  function 

iv.  Interoperability 

v.  Planned  ATD  or  ACTD  developments,  if  applicable. 

9.  Exit  Criteria 

Responsible  party:  JPEO-CBD.  Key  technical  measures  of  readiness. 

Identify  quantifiable  criteria  that  will  be  used  to  assess  effectiveness 
and  suitability  of  the  technology /product  development  effort. 

Provide: 

a.  Conditions  under  which  technology/ product  will  be 
tested/ demonstrated  prior  to  delivery  to  acquisition. 

b.  Current  performance  of  the  technology /product. 

c.  Minimum  acceptable  performance  threshold. 

d.  Desired  final  goal/ objective. 

e.  Estimate  of  the  transition  TRL. 

f.  Establish  criteria  for  development  of  the  receiver  operating 
characteristics  (ROC)  curve  and  spider  chart  prepared  for  the 
Assessment  Panel.  These  criteria  will  include  definitive,  complete, 
and  measurable  parameters  including  all  applicable  key 
performance  parameters. 


Attribute/ 

Parameter 

Current 

Minimum 

Threshold 

Objective 

10.  Program  Plan 

Responsible  party:  JSTO.  Show  major  activities/efforts  planned  for  the 
technology/product  development  with  milestones.  Include  both  S&T  and 
acquisition  tasks /elements. 
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ID 

Task  Name 

Y1 

Y2 

Y3 

Y4 

Y5 

Y6 

1 

T  aclf  1 

. 

; 

2 

1  aSK  1 

Task  2 

:  : 

3 

Task  3 

i 

. I 

4 

Task  4 

5 

Intearated  Capability 

11.  Transition  Program  Element  Funding 


Transition  PE 

FY03 

FY04 

FY05 

FY06 

PE  Total 

0604***N 

|K 

|K 

IK 

IK 

IK 

FY  Total 

|K 

IK 

IK 

IK 

IK 

12.  Signatures 


Acquisition  Project  Manager  Date 

JPM  XXX 


S&T  Project  Manager  Date 

JSTO  DTRA-CB  CAPO  XXX 
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Appendix  I  —  RDT&E  Budget 
Activities 


The  RDT&E  budget  activities  are  broad  categories  reflecting  different  types 
of  RDT&E  efforts.  The  definitions  are  provided  below. 

Budget  Activity  1:  Basic  Research 

Basic  research  is  systematic  study  directed  toward  greater  knowledge  or 
understanding  of  the  fundamental  aspects  of  phenomena  and  of  observable 
facts  without  specific  applications  towards  processes  or  products  in  mind.  It 
includes  a  scientific  study  and  experimentation  directed  toward  increasing 
fundamental  knowledge  and  understanding  in  those  fields  of  the  physical, 
engineering,  environmental,  and  life  sciences  related  to  long-term  national 
security  needs.  It  is  farsighted  high  payoff  research  that  provides  the  basis 
for  technological  progress.  Basic  research  may  lead  to:  (a)  subsequent 
applied  research  and  advanced  technology  developments  in  Defense-related 
technologies,  and  (b)  new  and  improved  military  functional  capabilities  in 
areas  such  as  communications,  detection,  tracking,  surveillance,  propulsion, 
mobility,  guidance  and  control,  navigation,  energy  conversion,  materials  and 
structures,  and  personnel  support.  Program  elements  in  this  category 
involve  pre-Milestone  A  efforts. 

Budget  Activity  2:  Applied  Research 

Applied  research  is  systematic  study  to  understand  the  means  to  meet  a 
recognized  and  specific  need.  It  is  a  systematic  expansion  and  application  of 
knowledge  to  develop  useful  materials,  devices,  and  systems  or  methods.  It 
may  be  oriented,  ultimately,  toward  the  design,  development,  and 
improvement  of  prototypes  and  new  processes  to  meet  general  mission  area 
requirements.  Applied  research  may  translate  promising  basic  research  into 
solutions  for  broadly  defined  military  needs,  short  of  system  development. 
This  type  of  effort  may  vary  from  systematic  mission-directed  research 
beyond  that  in  Budget  Activity  1  to  sophisticated  breadboard  hardware, 
study,  programming  and  planning  efforts  that  establish  the  initial  feasibility 
and  practicality  of  proposed  solutions  to  technological  challenges.  It 
includes  studies,  investigations,  and  non-system  specific  technology  efforts. 
The  dominant  characteristic  is  that  applied  research  is  directed  toward 
general  military  needs  with  a  view  toward  developing  and  evaluating  the 
feasibility  and  practicality  of  proposed  solutions  and  determining  their 
parameters.  Applied  Research  precedes  system  specific  technology 
investigations  or  development.  Program  control  of  the  Applied  Research 
program  element  is  normally  exercised  by  general  level  of  effort.  Program 
elements  in  this  category  involve  pre-Milestone  B  efforts,  also  known  as 
Concept  and  Technology  Development  phase  tasks,  such  as  concept 
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exploration  efforts  and  paper  studies  of  alternative  concepts  for  meeting  a 
mission  need. 


Budget  Activity  3:  Advanced  Technology  Development 

This  budget  activity  includes  development  of  subsystems  and  components 
and  efforts  to  integrate  subsystems  and  components  into  system  prototypes 
for  field  experiments  and/or  tests  in  a  simulated  environment.  ATD 
includes  concept  and  technology  demonstration  of  components  and 
subsystems  or  system  models.  The  models  may  be  form,  fit  and  function 
prototypes  or  scaled  models  that  serve  the  same  demonstration  purpose. 
The  results  of  this  type  of  effort  are  proof  of  technological  feasibility  and 
assessment  of  subsystem  and  component  operability  and  producibility 
rather  than  the  development  of  hardware  for  service  use.  Projects  in  this 
category  have  a  direct  relevance  to  identified  military  needs.  Advanced 
Technology  Development  demonstrates  the  general  military  utility  or  cost 
reduction  potential  of  technology  when  applied  to  different  types  of  military 
equipment  or  techniques.  Program  elements  in  this  category  involve  pre- 
Miles  tone  B  efforts,  such  as  system  concept  demonstration,  joint  and 
Service- specific  experiments  or  Technology  Demonstrations  and  generally 
have  Technology  Readiness  Levels  of  4,  5,  or  6.  Projects  in  this  category  do 
not  necessarily  lead  to  subsequent  development  or  procurement  phases,  but 
should  have  the  goal  of  moving  out  of  Science  and  Technology  (S&T)  and 
into  the  acquisition  process  within  the  future  years  defense  program 
(FYDP).  Upon  successful  completion  of  projects  that  have  military  utility, 
the  technology  should  be  available  for  transition. 

Budget  Activity  4:  Advanced  Component  Development  and 
Prototypes 

Efforts  necessary  to  evaluate  integrated  technologies,  representative  modes 
or  prototype  systems  in  a  high  fidelity  and  realistic  operating  environment 
are  funded  in  this  budget  activity.  The  ACD&P  phase  includes  system 
specific  efforts  that  help  expedite  technology  transition  from  the  laboratory 
to  operational  use.  Emphasis  is  on  proving  component  and  subsystem 
maturity  prior  to  integration  in  major  and  complex,  systems  and  may 
involve  risk  reduction  initiatives. 

Program  elements  in  this  category  involve  efforts  prior  to  Milestone  B  and 
are  referred  to  as  advanced  component  development  activities  and  include 
technology  demonstration.  Completion  of  Technology  Readiness  Levels  6 
and  7  should  be  achieved  for  major  programs.  Program  control  is  exercised 
at  the  program  and  project  level.  A  logical  progression  of  program  phases 
and  development  and  / or  production  funding  must  be  evident  in  the 
FYDP. 
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Budget  Activity  5:  System  Development  and  Demonstration 

SDD  programs  have  passed  Milestone  B  approval  and  are  conducting 
engineering  and  manufacturing  development  tasks  aimed  at  meeting 
validated  requirements  prior  to  full-rate  production.  This  budget  activity  is 
characterized  by  major  line  item  projects  and  program  control  is  exercised 
by  review  of  individual  programs  and  projects.  Prototype  performance  is 
near  or  at  planned  operational  system  levels.  Characteristics  of  this  budget 
activity  involve  mature  system  development,  integration  and  demonstration 
to  support  Milestone  C  decisions  and  conducting  live  fire  test  and 
evaluation  (LFT&E)  and  initial  operational  test  and  evaluation  (IOT&E)  of 
production  representative  articles.  A  logical  progression  of  program  phases 
and  development  and  production  funding  must  be  evident  in  the  FYDP 
consistent  with  the  Department’s  full  funding  policy. 

Budget  Activity  6:  RDT&E  Management  Support 

This  budget  activity  includes  research,  development,  test  and  evaluation 
efforts  and  funds  to  sustain  and / or  modernize  the  installations  or 
operations  required  for  general  research,  development,  test  and  evaluation. 
Test  ranges,  military  construction,  maintenance  support  of  laboratories, 
operation  and  maintenance  of  test  aircraft  and  ships,  and  studies  and 
analyses  in  support  of  the  RDT&E  program  are  funded  in  this  budget 
activity.  Costs  of  laboratory  personnel,  either  in-house  or  contractor 
operated,  would  be  assigned  to  appropriate  projects  or  as  a  line  item  in  the 
Basic  Research,  Applied  Research,  or  Advanced  Technology  Development 
program  areas,  as  appropriate.  Military  construction  costs  directly  related  to 
major  development  programs  are  included. 

Budget  Activity  7:  Operational  Systems  Development 

This  budget  activity  includes  development  efforts  to  upgrade  systems  that 
have  been  fielded  or  have  received  approval  for  full  rate  production  and 
anticipate  production  funding  in  the  current  or  subsequent  fiscal  year.  All 
items  are  major  line  items  projects  that  appear  as  RDT&E  Costs  of  Weapon 
System  Elements  in  other  programs.  Program  control  is  exercised  by 
review  of  individual  projects.  Programs  in  this  category  involve  systems 
that  have  received  Milestone  C  approval.  A  logical  progression  of  program 
phases  and  development  and  production  funding  must  be  evident  in  the 
FYDP,  consistent  with  the  Department’s  full  funding  policy. 
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